Faces from the Collective: Shared memory anyone? · Faces from the Collective: Shared memory...

Faces from the Collective: Shared memory anyone? of 26

Video title: Faces from the Collective: Shared memory anyone?Date: 3/31/06Description: Sam Ramji sits down with resident UNIX guru and Microsoft Architect Jason Zions to discuss the history of Windows and UNIX interoperability.

Sam Ramji: Hi, I’m Sam Ramji. I’m the director of the Linux and Open Source interoperability lab at Microsoft. Today I’ve got Jason Zions, an architect in the Core Infrastructures Solutions Group. Jason, thanks for your time today.

Jason Zions: Happy to be here.

Sam: Great. So the first thing that I want to know is how does a deep UNIX guru like you get to be an architect at Microsoft? What brought you here?

Jason: I came here along with some technology that Microsoft acquired about six-and-a-half years ago. I left Hewlett-Packard in 1993, went to work for a UNIX company in Texas. And after a couple years there decided enough of that, and I started a company with a couple of friends to build some technology that would let you run UNIX applications on top of Windows. Took that; built it. That technology became something called Interix. And in 1999 Microsoft acquired the technology and acquired me along with it. Kind of like getting a first-round draft choice.

Sam: Absolutely.

Jason: And that’s how I came to Microsoft. So I’ve been working in the UNIX and Windows interop space for about 10 years now.

Sam: Wow, so even past the acquisition we’ve managed to keep you around for...

Jason: Absolutely.

Sam: And keep you occupied and busily interested.

Jason: You know, I’m still a UNIX guy at heart. In a lot of ways, I’ve been working on UNIX standards with the POSIX committee since 1988. Been involved for a very long time in that. Still have my finger in that space, but I love Windows; I love what Microsoft does; I love the things we do here. So there’s still lots of work for me to do.

Page 1 of 26

http://port25.technet.com/archive/2006/03/31/16.aspx


Sam: That’s really interesting.

Jason: It is.

Sam: I get surprising comments from people outside the industry, people working in UNIX, working in Linux saying, you know, “We don’t understand how we can work with Microsoft. How can all this stuff fit together? Why would Microsoft even have a UNIX strategy?”

Jason: And that’s what I love about being here is that it is my job to spend so much time looking at how can we make UNIX and Windows fit together? There are customers who’re always gonna have some of everything. How do you make it work? That’s what my focus is. That’s what a lot of what I do for Core Infrastructures Solutions is figure out how those pieces fit together.

Sam: Yeah, there’re some core challenges there, in older systems and today, ongoing. So you must be doing a lot of work in driving those architectures forward.

Jason: Absolutely.

Sam: So let’s talk a little bit about the history for the Services for UNIX in Microsoft Windows. As you said, you built Interix. That was probably way back in the NT 3.5 time frame.

Jason: Yep.

Sam: Why don’t you talk us through that?

Jason: We built the technology which was originally called OpenNT back in 1995 and ’96. The product shipped in ’96. We put out four or five versions over the next three or four years. Ran through Windows 2000. By that time, we’d been acquired by Microsoft. The technology came into the company. Microsoft took that Interix subsystem and combined it with the other interoperability technologies that they had in what was the original Services for UNIX product, put it all together, and that’s what I spent a couple years working on integrating. That technology, Services for UNIX, had a 3.0 release and a 3.5 release. And now those technologies, in a much more enhanced and better-integrated form, appeared directly in Windows Server 2003 R2.

Page 2 of 26


Sam: Interesting. And there is a trend of that carrying forward into Vista under the SUA name, if I recall it correctly.

Jason: Well, the UNIX environment is called SUA, the Subsystem for UNIX Applications. That appears in R2, and it also appears in Vista.

Sam: All right. I’d like to talk through the architecture of that system.

Jason: Sure.

Sam: One of the things that almost knocked me out of my chair the first time I heard you could run UNIX applications on Windows, and I thought, how would that actually happen?

Jason: Yeah, yeah.

Sam: And finally I came across a diagram that I think you built, which made it all clear, so….

Jason: Yeah, architecturally, if you look at the way the Windows operating system is built, you’ve got at the bottom, the hardware abstraction layer that isolates the design of the actual hardware—its BUS structure and everything else—from what the operating system kernel itself has to care about. Then you’ve got the kernel, device drivers, the usual things living in the kernel. On top of the kernel, the way Windows is architected, are these subsystems. Some are functional subsystems, things like LSASS, Local Security Authority...

Sam: And GGI, and all this stuff you’re used to.

Jason: Right. And others are environmental subsystems. So you’ve got the Win32 subsystem—that lives partly in the kernel but mostly a lot in user mode—that provides what looks like the user interface for the Windows operating system. And it’s really just a personality built on top of that kernel. But what you also have with SUA, is you have a completely separate stack that also sits on top of that common underpinning. And that stack actually gives you the UNIX kernel. The pre UNIX kernel functionality gives you the UNIX utilities, the user mode, let’s you run X11 applications—all those things living on top of that same stack, parallel to the Win32 stack. They actually share some functionality underneath. For example, the sockets functionality living in the subsystems of UNIX

Page 3 of 26


applications ties directly into the Winsock functionality of the Win32 world. So you don’t have two separate IP stacks or two separate IP addresses. It’s one networking infrastructure, two interfaces, two user programming modes.

Sam: That’s pretty wild if I think about that: Since they both depend on the same kernel, I could have shared memory between a Windows and a UNIX application all running on one Windows server.

Jason: Absolutely. That’s worked in the subsystems of UNIX applications ever since we added the shared memory functionality four or five years ago.

Sam: So what’s the wildest use for that kind of sharing of memory or maybe socket communication between fully active programs on the Win32 stack and the UNIX stack at the same time?

Jason: Well, there’s been a couple of things that I’ve seen customers do. I know that there was a customer who had a UNIX application that basically produced a whole bunch of textual output, but what they really wanted it was to be able to wrap it in some sort of wrapper and reuse it as a COM object.

Sam: Sure.

Jason: So they tied the two things together, took the text, parsed it up, and communicated through COM. There’re applications that’re built that share information directly through a shared memory section to really make things work better with very high performance transport. There was one customer today that moved a million lines of stock UNIX source code in a weekend. Now they spent four weeks to test the living heck out of it. But the code got up and running in the course of a weekend.

Sam: So they took a giant C application that was running perfectly happy on another UNIX box that were...

Jason: It was a whole family of applications that in total was about a million lines of code. I know of a guy....

Sam: That’s not trivial.

Page 4 of 26


Jason: It’s not trivial, but the environment was good enough and close enough to the typical average UNIX system that the kinds of problems they ran into were the same kind of problems they ran into in moving from one version of Solaris to the next. Or from Solaris to HP-UX or to AIX. Actually smaller problems than to moving to AIX, but let’s not go there.

Sam: Yeah. The cautionary tales from the experience in porting any other UNIX application to any other interface—you can do it here, too.

Jason: The more often you’ve ported in some particular chunk of code, the more easily it’s going to move to the SUA environment.

Sam: So you keep stripping away the nonstandard pieces.

Jason: And you find standard idioms for doing things that you might’ve used a platform-specific functionality for when you first wrote the code.

Sam: What led you to put a POSIX environment on Windows in the first place?

Jason: Well, the original impetus for Microsoft architecting Windows that way goes all the way back to a U.S. government directive. It’s called FIPS 151. This was a rule that the U.S. government put in place for acquiring computer systems that said there’s the standard operating system called POSIX, and every computer system we’re gonna buy has to have a POSIX interface.

Sam: So it was a check box...

Jason: It was a check box.

Sam: In order to get Microsoft Windows bought by the government?

Jason: When N2 was being architected, the architects knew that there was this requirement. And they said, “Well, I guess we’ll have to provide this.” And they designed Windows in such a way that you could put this POSIX interface on top of it. Now at the same time they were architecting NT, there also was this other operating system well-known in the business called “OS2.” So when Windows was designed, it was designed to host multiple personalities. You had the Win32 personality; you had the OS2 personality;

Page 5 of 26


you had the POSIX personality. Over time the OS2 personality dropped out of favor because the OS2 market vanished. The POSIX personality started originally as a check-box item. Over time, people actually wanted to make it real. My involvement in this dates back to 1994 or ’95, when the U.S. Coast Guard put out a bid for an operating system to serve their needs, and their bid specified POSIX.1 and POSIX.2. POSIX.2 is the standard that defines the commands, the utilities, the shells of all the utilities, awk, grep, Perl, sed….

Sam: So that you know as an administrator that you have the environment that you’re used to relying on...

Jason: Exactly.

Sam: And you can take your scripts and migrate them.

Jason: So Microsoft had the POSIX.1 environment that was a check-list item, just good enough to pass the tests but not really good enough to use. But they also had this requirement from the Coast Guard to put the utilities ______ on it, which required something actually industrial strength.

Sam: Absolutely.

Jason: That’s why we got into the business—myself and a couple of other friends—and formed this company to take what Microsoft had and really build on top of it and make it good enough to host those utilities, get the utilities up and running. So we started by taking 44bsd light source code. Hoarding those utilities on top of the POSIX subsystem that Microsoft shipped, the NT 3.51, and the NT 4, and that was the very first version of the product we built.

Sam: That is pretty cool. And that was the product called Interix.

Jason: That was the product at that time was called OpenNT.

Sam: Okay.

Page 6 of 26


Jason: And after a time, Microsoft started reminding people that NT was a trademark, and we couldn’t just use it. So we had to change the name of the product, and we called it “Interix.” That was around the 2.0 time frame a couple of years later.

Sam: It sounds like more than just creating a check box—you know, POSIX.2 satisfying a solution. You actually went for user acceptance. You said, “I want to be able to have an admin who’s just completely comfortable and loves this as much as anything else.”

Jason: My goal is I’m a UNIX guy. I wanted as close to a UNIX–compliant operating system as I could get. I’d been participating In POSIX for years—a decade by that time. I wanted something that I could take UNIX code, put it on this platform, and it was just gonna run. Now we started with the basics. All the utilities. And then we moved onto things like the X11 application environment. By the time you add in all of the components that appear in SFU 3.0 or 3.5, the UNIX environment is about three-and-a-half million lines of code. A vast majority of which originated in the open source environment. Mostly BSD licensed.

Sam: Yeah, that’s a good, permissive license.

Jason: And we needed an operating system, the subsystem itself—the Interix subsystem at the time—and now SUA that was rich enough and strong enough and robust enough to support all of that code and to be as secure as the rest of Windows. More secure than the rest of Windows.

Sam: Now you say “more secure,” which is an interesting distinction.

Jason: Well, in a lot of cases, we tried to tie things down and lock things down very tightly. There’re places where we had to step out of the Windows security model. So for example, UNIX has setuid capability, where running an application by virtue of the way the permissions were set on the app, the application when it’s running takes on an identity with more permissions. Windows doesn’t have that notion. It can impersonate another user, but just by virtue of running an app, it doesn’t give you any more permissions. We needed to add setuid capability so we’d actually have full UNIX compatibility. We needed the standard UNIX 9 mode bits. Map that down on top of the rich access control list environment that NTFS supports. So we mapped those mode bits onto everything. But at the end of the day, we had to make certain we had no security vulnerabilities, that we weren’t crashing. You know, blue screens are bad, whether it’s just the UNIX environment that goes down or the whole box that goes down. And living on top of NT 3.51 and NT 4

Page 7 of 26


as we were when we first wrote the code, we were shooting to be as rock solid as we possibly could be.

Sam: So you wanted to be as solid as the kernel.

Jason: Sure. That was our goal: be eventually not a single bug against the subsystem itself. Yeah, there’re gonna be bugs in the applications. There’re bugs in everybody’s applications.

Sam: Absolutely.

Jason: But the NT kernel today doesn’t have bugs. You don’t actually see security defects being filed against the NT kernel. They’re not being found; they don’t exist. That’s probably some of the best-reviewed code on the planet.

Sam: Yeah, absolutely. More workloads, more eyeballs. Would it make sense to mount a server, let’s say 2003 R2, and would you be able to not initialize the Windows subsystem and only have SUA running and run SUA applications? Is that a sensible thing to have?

Jason: Well, there’s an awful lot of the Windows infrastructure that is built on top of Win32, so it’s gonna come up, even just the login screen. You give the three-finger salute, you get a login box. That’s GINA, which lives on top of LSASS, which lives on top of a great deal of the rest of the Win32 infrastructure. So you bring that up to log in, but you can configure a Windows machine so that when you log in what you get is a KornShell prompt. You get a command window; you get a nice blinking dollar-sign prompt, and that’s your life.

Sam: So I could run this pretty lean if I wanted to...

Jason: You absolutely could.

Sam: Especially if I wanted to take over. Let’s say I’m a large retail company, and I wanted to move over my pricing core algorithm, right? Which is sitting around in C. I’ve got some archaic ways of getting to it. I want to open it up to more different kinds of usage. The first thing I want to do is bring it over, run under SFU or SUA, depending on what release I’m using.

Page 8 of 26


Jason: And moving onto R2, that code, especially if it’s been moved to multiple UNIX variance, is generally going to come over pretty easily. It’ll just build right up and run under SUA, and the code is just gonna function as is, no changes. Now if there are third-party libraries that you’re using, you might have to get Windows versions of those and then interface between that code and the third-party versions. So a typical example there is you’ve got some fairly large application that lives on top of Oracle. The app was written to the Oracle call-level interface, OCI. So you’d take that app, you’d move it into SUA. You take OCI, and you tie to the Win32 OCI.DLL library. And then you’d be using that infrastructure to make your OCI calls directly out of the app.

Sam: Would you have to make any modifications to your UNIX application to be able to invoke the OCI DLL?

Jason: With SUA and R2, we actually provide out-of-the-box direct hooks for OCI and for ODBC. So we take care of that for you. For other libraries, yeah, there’s some engineering work you’re gonna have to do. You’re gonna have to accommodate differences in the signal model. The asynchronous notification model between UNIX, which is signal based, and Windows, which is event based. So if you’ve got a Windows library that’s expecting to deliver asynchronous notifications by events, you have to translate those into signals. But it’s a fairly simple and fairly repetitive kind of engineering work to do the wrapper yourself.

Sam: Do we have some best practices around that? Is that some of the stuff that you’re working on?

Jason: Those’re being evolved. We’re working on writing down how does this all work? What’s the best way to do it? I’ve found that in my work, the way I typically like to do things is I’ll use separate threads. I’ll have a thread running the UNIX code and a paired Win32 thread that’s actually making the call to Win32 library.

Sam: Listening for events, sending signals.

Jason: And very carefully controlled throwing—exactly. Catch the event here, and throw the signal over here. Very carefully control how these things work.

Sam: It’s like a buffer.

Page 9 of 26


Jason: As a programmer, it’s real easy in my mind to keep the paradigm separate that way, to really control things well.

Sam: That seems like a good pattern to deal with the impedance mismatch between the two models.

Jason: Mmm-hmm.

Sam: That makes a lot of sense.

Jason: Yep.

Sam: So when people start using SFU, actually real people out in the world, you know, whether it’s a….First of all, I guess my first question would be are UNIX administrators comfortable in this environment? Is this a place where they bring this up and they feel fairly at home, or is there some mental gymnastics they have to go through to orient themselves?

Jason: As a UNIX admin, you want things like NFS to work. And you want to be able to use the NFS command that you’ve always used. So you want to be able to use mount, you want to be able to use export. And those commands work. When you look from UNIX system to UNIX system, there’s tremendous variation, even between Solaris and HP-UX. So as an admin dealing with multiple platforms and multiple sources, you already have a certain amount of flexibility in the way you expect these utilities to look. So adjusting to the ones provided by R2, with the NSF client and server functionality, or the stuff that’s available in SFU, the commands are very similar in the way that they look and feel. Once you bring up a KornShell window, you’ve got exactly all the things that you expect to see. You’ve got ps to manage which processes are running. You’ve got kill to turn to them off if you need to. You’ve got all the typical command-line utilities you’d expect to see for moderating a UNIX environment or building scripts that actually do things to connect applications together. A lot of this stuff...

Sam: Where you’re pricing inputs and outputs and pricing text and getting ________....all that stuff…

Jason: Exactly. A lot of the stuff just works right out of the box. Now, again, those places where you see tremendous variation in interface between UNIX systems—where Solaris is tremendously different from AIX is different again from LINUX is different again from

Page 10 of 26


HP-UX—in those cases, there are yet again command-line tools in the Windows environment that will let you do the same things. You just have to make the same changes. While many of these may be Win32 command-line utilities, but the beauty of the SUA environment is you can just run a Win32 command right from the KornShell. You know, throw an alias on top of it to hide the .exe. There’s a lot of things you can do—put things in your path. There’re mechanisms to make it real easy to just invoke a Win32 program. You just pipe things right into it. You parse its output coming right back out, just like on any other UNIX machine.

Sam: So with all that functionality you’ve got, basically for administrators and for applications—but let’s focus on the admin for a minute—an admin would feel right at home with their ability to do KornShell and bash and all this other stuff. They’ve got access to the whole Windows kernel, all of the UNIX commands, the UNIX apps, and all of the Win32 command-line interface. So why would we also be building something like Monad? What’s my advantage there, and what would I be looking for?

Jason: I think Monad is really cool stuff. When we started thinking about what to do about a new command-line interface for the Windows platform, we took a look at what people liked in the UNIX world, what they liked about UNIX scripting, which was independent tools that did fairly well-defined things—maybe a bunch of well-defined things but very cleanly independent—that you could plug together by piping between them that had a pretty well-defined execution environment, pipelines for connecting things, paths to flow data around. That was what people liked.

Sam: It’s automatable, right? I mean...

Jason: It’s beautifully automatable. The other things that’s neat about that environment is your normal command-line interface is also your scripting interface. It’s the same commands.

Sam: Exactly.

Jason: So getting good at scripting naturally made you a better user of the command line for your daily interactive work. Being really good at the keyboard for doing interactive work naturally made you a better scripter.

Sam: As well as getting you off the programming path, too. I mean, I’ve seen knock-down, drag-out arguments between Windows admins and UNIX admins about who’s got more skills and who can do what. And really it comes down to, you know, it’s really cats

Page 11 of 26


and dogs, right? You’ve got people who’re GUI admins and you’ve got people who are shell admins, and never the twain shall meet. But it looks like it’s starting to meet.

Jason: Now, what we knew was that there were some things that are a lot easier to do from a GUI. But there’re other things that are really a lot easier to do from a command line that the Windows environment used to make hard to do from the command line. CMD is a lousy scripting environment. Before Windows Server 2003, especially before Windows 2000, there were so many admin things you could only do through GUI. They were just completely not doable from the command line. With Windows Server 2003, there was a huge push on to make sure that everything that was manageable could be done from the command line. So that addressed one part of the problem. But the other part of the problem is CMD is just really a tough environment to build scripts. So we need a better scripting engine. We looked at what people loved about UNIX scripting, which, as I said, was these independent, composable tools, with the pipeline to everything else. But we also listened to what they hated about it. And this resonated really well with me, who’s been writing shell scripts for 20 years. I hate the fact that I’m always parsing text. I run some new command, and I pipe it into something that cuts out a couple of columns of text. And then I run into another utility that does some filtering based on range, and that filter starts by taking that text and converting it to numbers, then doing its filtering, and then it spits out more text by converting the numbers back to text, on to the next thing in the pipeline, and so on. And every time you tear apart a command line, a line of data in the pipeline, you’re throwing away information that you might need later.

Sam: You’re spawning processes.

Jason: You’re spawning processes, all these other things. So that’s the downside of the UNIX scripting environment.

Sam: So it’s basically lossy every time you want to go from one script to...

Jason: Right. So in reimagining this, we thought, what we want is the good stuff—this rich environment surrounding it, ultracomposable tools, the notion of pipeline. But what we wanted to get rid of was all the text parsing. This was all being thought of at the same time that .NET was starting to become really a real thing inside Microsoft. Where you had WMI, which was this object infrastructure for management information about Windows. You had Exchange with its own object model. You had Office—so many Office applications with their own object models. But .NET was designed to provide access to those object models. So we thought, wouldn’t it be really cool if you could have a scripting language that natively lived inside this sea of objects and let you get at any one of these data sources and natively manipulate the object that came out of them? So that instead of

Page 12 of 26


having to take an Excel spreadsheet and convert it to flat text and, say, a comma-separated file, a .csv file, and tear apart the column to do some parsing, if I could just have an object provider that let me extract certain objects directly out of the Excel spreadsheet as objects?…

Sam: Walk the object tree, invoke them as necessary.

Jason: Right. Get directly at the object model so I can invoke methods directly on the objects, pull out attributes of the objects themselves and directly look at data values, and never have to convert back and forth between texts.

Sam: So now you’re doing scripting and invocation as opposed to text parsing.

Jason: Exactly.

Sam: That makes a lot more sense.

Jason: And you swim in the sea of objects, and the more of Windows that becomes enabled to this—you’ve got things already like the kernel being instrumented with WMI—it means I don’t have to read /dev/kmem the way I do in the UNIX world. Or I don’t have to run ps and parse the output, and ps is different between every UNIX system. You’ve got some things that’re the same, some are different. Where the columns appear depends upon how wide the data values are. There’s so much variation on that particular space that getting at… in the scripting world, system-level information is really obnoxious in shell scripts. It’s tremendously easier in Monad because you’ve got WMI, which there’s a WMI provider that lets you get objects right out of the Windows kernel that reflect all of this information. Using the reflection capability of the .NET environment, you can actually get an object and ask it what it is, ask it what methods it has: “What can you do? What can I tell you to do? What data values can you provide to me? What type is that data value? What are the names of those things?”

Sam: There’s something you can ____ play around and explore with the command line, and then you can write very sophisticated scripts if you wanted.

Jason: Exactly. I don’t have to dig out the man page, I don’t have to dig out a CBL table, I don’t have to read a book. I can actually say, “Why don’t you give me the system status objects?” and I can ask what those are named. Look for a name that seems pretty obvious, ask it what it’s data elements are. And those are named in a pretty obvious way.

Page 13 of 26


And now I can get half the information I want in a script with great ease, and I still have the full object flowing down the pike. So I’ve written the script that does something for me today—great! In three months I come back, I realize there’s some other piece of information I want to extract. Now instead of having to go back to where I had cut a few columns out and add one more column back in—and pray I got that right, and fix my parsing all the way down the pipeline—that data was always in the object, and it stayed persistent data all the way through. I have access to it anywhere else. Much richer environment for building.

Sam: So you never throw away ______________ personal code.

Jason: And all the parsing, which is where my errors always crept in, all go away.

Sam: Well, the less code you have to write, you know, the less errors you’re going to make.

Jason: Exactly.

Sam: Well, plus debugging the scripts, you know, print-based debugging. Not the most pleasant ____....

Jason: Yeah, it’s harsh.

Sam: But speaking of fun, I was thinking about what kind of applications can you really register, see how UNIX—see how it really is. I heard you imported a UNIX version of Doom and ran it on a SUA.

Jason: Yeah. This is something that the SUA developers were doing sort of to amuse themselves but also to make sure that they got it right. Again, you want to look for this rich set of utilities to make sure that you’ve got an environment accurate enough to run them. So they took the UNIX version of Doom. Got it up and running on top of SUA. They put an X server on the workstation, and they ran Doom.

Sam: That answered another question I was going to ask.

Jason: It all just works. Yeah, SUA, the R2 stuff does not include an X server. You’ll need to get one if you want an X server. The client functionality is in there, so you can build X

Page 14 of 26


applications. You’ve got all the X tools. You’ve got all the X client applications—things like xterm and everything else like that. What we don’t include is an X server. A lot of customers already have one. For customers that don’t, there’re a dozen to choose from. You know, what additional value would Microsoft have by putting our own in the box, and then...

Sam: But any of those just run ahead and _______....

Jason: Yeah, any one of them will work just fine. There’re two or three that I particularly like, but they all work. You know, pick one. The open source one works.

Sam: Outstanding.

Jason: You know, it all just does the thing. So yeah, they put in X-Tetris, which I thought was kind of amusing. I’m a Tetris kind of guy. But I realized after playing the test version again for a while, oh, my carpal tunnel’s coming back. Where’d that come from?

Sam: I know, I know. I can still see those shapes when I close my eyes.

Jason: Oh, I can’t play too long, because then I have weird dreams.

Sam: It’s really bad.

Jason: Weird dreams.

Sam: You talked about open source. I want to interrupt you for a second. So how comfortable would a Linux admin be working on SUA? I mean, is that basically as similar or as different as Linux versus any of the other Unices?

Jason: It’s pretty much the same kind of delta, where you’ve got a difference between the Linux tools and Solaris tools. But the thing is if there’s some Linux admin tool that you really happen to like that doesn’t exist under SUA, import it.

Sam: How hard is that to accomplish that part? Do you have to have a lot engineering knowledge?

Page 15 of 26


Jason: It really varies on just how platform specific the tool may be. So if you have some customized tool to manage automounts, for example, the automount infrastructure is very system specific. So that one you’re gonna have a really hard time moving over. And in fact, the whole Windows infrastructure and architecture for automounting is radically different. You don’t actually have an automounter to configure as such. You have UNC naming, where it’s just whack-whack, host-whack, share-whack, whatever, and you have access to it without having to do any automount configuration at all. What you don’t have is hiding of what that main space looks like on some arbitrary local set of names.

Sam: I see.

Jason: So you wouldn’t really rely on the automounter if you didn’t have to. It’s one of those differences that people have to sort of get used to as they add Windows machines to a UNIX environment. They used to say, “Great, the engineering team gets this view of our corporate data hierarchy, and the accounting team gets this other view, and we’ll use the automounter to construct that.” But what you now have with Windows and UNC is there’s one true view of the name space. Now you have DFS, which lets you get replication and a bunch of other things, but it’s still—from the UNIX perspective under SUA—it’s slash net, slash host, slash share, slash whatever. That’s the way in a SUA app you’d refer to a UNC name. Whack-whack host, whack share, whack whatever.

Sam: Got it.

Jason: So we have that infrastructure all built in.

Sam: So in effect, I can either run a number of those applications or import them if I need to.

Jason: Yep.

Sam: And then it can be proportional to how much system-specific code I’ve got. That makes a lot of sense.

Jason: Yeah, there’s a Web site out there—it’s www.interopsystems.com—that actually has a fairly large library of open source apps that they’ve ported to the SUA environment.

Sam: That’s excellent.

Page 16 of 26

http://www.interopsystems.com/


Jason: You know, things like the GIP. You have three million lines of code by the time you flatten out all of the library dependencies. And it took them a while to get exactly the right version of the libraries…

Sam: That’s a challenge.

Jason: …to make those available. So we’ve all of a sudden got about three million lines of code, and they have a list of all the changes they had to make to that code to get it up and running. And it’s pretty small.

Sam: That would be interesting to look at.

Jason: Apache 2. There are two code changes required to the Apache 2 source base to get it up and running on top of SUA. And they both really revolve around a key distinction in the security model between UNIX and Windows. In UNIX, and it’s all around setuid again. Remember, with a setuid application you can adopt an arbitrary identity. If you’re a super user, you can become anyone. So amongst other things, you can actually adopt any list of groups as your supplemental groups. So there’s this API that some UNIX systems support to allow you to construct your arbitrary group identity. And different UNIX systems have a different API to do that. But they all have one or the other. You can’t do that in Windows. All you can really do is adopt a particular identity.

Sam: Right ______________________.

Jason: Exactly correct. So there’s no compile-time IFDEF to say, “I don’t have any of those. Live without it.” And that’s the code change you have to inject is basically an IFDEF that says, “Oh, I’m in this environment; don’t even try that trick.”

Sam: Sure.

Jason: But you don’t actually need it. Very small number of changes.

Sam: So that brings up a question about the usage of SUA. Why would an engineer organization start using SUA? My initial take is that you’ve got tons of C applications running in UNIX—you know, whatever language you choose to pick—that’s running a lot of the corporate workload, whether it’s batch applications or however they’re interoperating or how they’re functioning. I want to bring those over to an x86-based

Page 17 of 26


infrastructure. I want to start taking advantage of maybe more-modern GUIs. What does that process look like? Can you walk through a little bit some of the customer concerns and then a little bit about how you actually do that, since you’re actually writing the solutions guides to this process?

Jason: In a lot of cases, a decision is made to start moving to Windows, and, you know, there’s a lot of business reasons why that’s a good thing. There’s total cost of ownership; there’s cost of acquisition issues; there’s all the additional technical capabilities that the Windows platform provides. And if you’re gonna add a new system to your environment, you naturally begin to think, how can I consolidate onto that newer platform? Are there things that I can move where I can turn old boxes off? As old UNIX systems come out of support life, as they come off lease, there’s this business imperative to find a new home for them.

Sam: Oh, I see.

Jason: And if you’re going to be bringing in new Windows systems, why can’t that be that the new home versus going out and spending two or three or five times more money on a newer UNIX box or running some of the TCO risks and all the other things that are associated with bringing in Linux systems? You could, in fact, choose to put them on the Windows machine.

Sam: Especially with these older scale-up architectures, as opposed to what you can do with scale out.

Jason: Exactly right. So your first natural inclination is, well, why don’t I just take the code, put it down there and try to build it?

Sam: Sure.

Jason: Move the source code over, hit make, and see what happens. And for a lot of applications, there’re two or three things you have to diddle in the make file to accommodate for differences in where library files live in that. Once you get the make file done correctly, most of the app has just got to build. There’re gonna be three or four or five portability things. You’re probably going to have to fix new code someplace. When you get those all done, you’ve got the application up, and it’s running. Now the first thing you do is test the living heck out of it, make sure you haven’t broken something subtle.

Page 18 of 26


Sam: Absolutely bang on it.

Jason: But that’s step one. So step one is you’ve gotten from one platform to another with the same functionality, the same application, the same code. So you’ve earned the benefits of getting it through a platform, but you haven’t yet gotten any leverage out of that newer platform.

Sam: Right, so now I’m running all this stuff on a Windows kernel on a Windows box.

Jason: And the upside is all the knowledge that your devs had about what the code looks like—100 percent preserved.

Sam: Totally intact.

Jason: Dev skills 100 percent preserved. Admin skills—using the command-line environment that SUA provides with the KornShell and everything else, the ability to wrap around a bunch of Windows tools—many of those admin skills come over 100 percent preserved. A lot of your scripting for doing multisystem administration, if you don’t choose to move the things like using group policy to manage systems as groups, you don’t want to stay away from the GUI thing, you can still do it the hard way in scripting. Those skills transfer. But now you’ve got the new platform, no leverage of it other than just getting up and running. So that gets you some of the business value. The next step then is, how do I leverage the other things that the Windows platform brings to me—the reason I went to it in the first place? Maybe it’s faster development using Visual Studio, maybe it’s the initial capabilities that the .NET framework brings me for free, things like XML and SOAP.

Sam: Yeah, I hear a lot of customers wanting XML enable, you know, archaic or legacy applications. And when we look at legacy, and some people think it’s got a bad word, it’s a bad word. Legacy means it works!

Jason: It’s the crown jewels of the company.

Sam: Those are the crown jewels.

Jason: Absolutely.

Page 19 of 26


Sam: Your pricing engine. Right, this is how you do your financial derivatives, right? But you still want to be able to extend that. Let’s say you’re a financial services company. You want to be able to extend that out…

Sam: Exactly.

Sam: …and be a consultant in a remote office. You want to use a Web architecture, and you want to take all this stuff and throw it through an XML ____....

Jason: The beautiful thing about this approach is once you’ve moved the code over and it’s on the Windows platform, using SUA you have the ability to extend bits and pieces of it as you choose and as your schedule permits. As the demands on the application grow, you can change bits of it to be written in C# or in managed C or in managed C++, living in the .NET framework, which gives you all of the capabilities that the .NET framework brings you.

Sam: So security and memory management and all that stuff.

Jason: Its native libraries to handle XML, its much simpler exposure as a Web service.

Sam: You can get WCF, right? You’d be able to use the indigo subsystem and then….

Jason: All this really cool stuff for free that comes, and it all just works, and it’s all nice and tightly integrated and everything else. And you can begin to use it at your own pace. You rewrite bits and pieces of the application to use them as you choose to. Using the integration capabilities of SUA, everything from direct Win32 calls within your SUA application to shared memory to the common file system that lives underneath everything to semaphores—all the ways that these things can communicate. And you can architecturally structure an application that over time has less and less code because the framework and the libraries take on more and more of a load.

Sam: Well that’s the key. I’ve worked in companies before where we had a cross-platform framework. And then you build it. And it’s always fun. It’s pretty sexy to build a framework the first time, doing all this new object-oriented stuff.

Jason: And the maintenance stinks.

Page 20 of 26


Sam: Maintenance, you know, the architect leaves and then you end up sending some junior devs in to fix the parts of the system that they didn’t understand.

Jason: And they make it worse.

Sam: And the whole thing just kind of gets out of control.

Jason: Exactly.

Sam: So the idea of being able to get off that framework, I think that’s pretty attractive from just a… what is the cool stuff to do these days? I think it’s really more writing applications at the XML layer and figuring what’s my message path gonna be, and how am I gonna process this stuff? And not, ____, how do I take XML? I know one company that builds an XML parser in COBOL.

Jason: Oh, boy!

Sam: From scratch!

Jason: Yahoo!

Sam: Just to be able to say they’re building their own.

Jason: Oh, man! That’s… are you serious?

Sam: It wasn’t—it was a Fortune 100 financial services company.

Jason: Yikes! Somebody had more money in their budget than they knew what to do with.

Sam: Not a great use of company resources.

Jason: Yeah. That’s funny. The thing is when you treat operating systems as commodities like interchangeable parts, what you wind up with is settling for functionality at the commodity level functionality. Or that least common denominator that they all have.

Page 21 of 26


If you actually pick an operating system and leverage its benefits, leverage its capabilities, use what comes along with it, it’s not a commodity; it is differentiated; there’re things that Windows does better than UNIX. And, you know, I’m gonna be really honest, I’m sure that my management will shoot me for saying it, but there’re probably a couple of things that UNIX does better than Windows.

Sam: Absolutely.

Jason: And you know something? That’s okay. The reason that we put all of this UNIX compatibility and interoperability stuff into Windows for the long haul—it’s in R2, it’s in Vista, it’s going to be in Windows moving forward—is we know full well, customers will always have UNIX systems.

Sam: Well, it’s a heterogeneous world. And that has always been the vendor strategy...

Jason: Exactly!

Sam: if not the customer strategy.

Jason: Nobody does forklift upgrades. It doesn’t happen.

Sam: Leave in layers is the customer strategy. How do you do that and work with what’s there?

Jason: So SUA lets you layer on top of this core code that you already have, that already works, that you understand—the crown jewels—but it lets you strip off the stuff that isn’t the essence of it. So you’ve got this pricing application, and it’s wrapped around—it’s wrapped inside this X GUI, and it’s sort of okay, but let’s face it: It’s 1980s, 1990s design. It’s kind of crusty. And the people you’re hiring today, the kids you’re hiring today—the 20-somethings, the 30-somethings—they’ve grown up with Windows. And they’re used to a Windows user interface, and there’s so many great things you could do if you had the ability to use Web services to combine the interfaces of multiple applications together at once

Sam: That’s where it starts to get really powerful.

Jason: Using Win Forms Web Forms.

Page 22 of 26


Sam: Yeah.

Jason: So you could strip off that surrounding X stuff, and build something using Visual Studio with Win Forms or Web Forms to rapidly prototype a really usable modern interface but still have the untouched UNIX code living in the middle. That’s power. That’s leveraging the investment that you’ve already made as a business. There’s real dollars in there. There’s millions, if not billions, of dollars invested in that. A lot of people run their companies on it. Financial services companies.

Sam: Absolutely.

Jason: This is what they do.

Sam: It’s known good.

Jason: Exactly. You don’t touch it if you don’t have to.

Sam: Absolutely.

Jason: So you start with that, wrap around it the user interface code that everybody knows Windows does better.

Sam: Now when I build that, I’m moving from a VI and Emacs world, right? I want to be able to run Visual Studio, program things in Visual Studio. Can I compile and make and execute things under SUA from VS and debug it?

Jason: Visual Studio 2005…

Sam: And debugging is really the dirty part of all this stuff.

Jason: Yeah, it sure is, isn’t it?

Sam: And there’s lots and lots of debugging to do.

Page 23 of 26


Jason: Yeah, Visual Studio 2005 does a couple of things for you. First of all, it will let you build SUA applications right from there. There’s a plug-in module you can download that will actually allow Visual Studio to directly debug SUA applications. But even if you don’t like Visual Studio, if you really are an Emacs kind of guy—and for a lot of years I was—I mean that was wired to my fingers, and I wanted to make my apps inside of Emacs, and I wanted to use GDB to debug them. GDB works, too. Pick your poison. You want to use GDB? Go, do it! You shouldn’t have to choose. Why does it have to be an either-or thing?

Sam: Shouldn’t be.

Jason: Should be both apps.

Sam: Absolutely.

Jason: You want UNIX applications? Great! You can run them on top of Windows, and leverage all that Windows brings to you. Want Windows applications? Great! Run them on top of Windows. We do that better than anyone else does.

Sam: That’s awesome.

Jason: And the operating system was designed from day one to support this notion of multiple personalities. All that we’ve been doing over time—and culminating in the SUA work that appears today in R2 and will appear in the Vista client—is really tightening up and improving the coupling between those two worlds. So that between the UNIX world and the Windows world, on top of that common operating system, you have the ability to construct applications that live in both worlds.

Sam: And this really goes back to the core work that you’re doing now in core infrastructure solutions. It’s really bring the worlds together and have best practices to understand exactly how to get this stuff done.

Jason: How do you move applications onto the Windows platform, whether you’re doing it by running them under SUA or you’ve got that UNIX code and you want to move it directly to Win32? We’ve got that UNIX code, and you want it to live in the .NET environment as managed C or managed C++ or even C#. What’re the best practices for moving that code over? We’ve got guidance development work going underway right now. I’m actually probably late for reviewing…

Page 24 of 26


Sam: I’ve done that….

Jason: …one volume in that particular guide. And that’s all cooking along, and we’re gonna deliver that in the next couple of months. But that’s all based on actual practice, what people really do in moving applications. It’s not pie-in-the-sky, bunch of developers sitting in an office, going, you know, “How would I do this if I had to?” No, no, we talked to people who actually do it for a living: “How did you solve that problem?”

Sam: And as the developer on the administrator side of the whole package.

Jason: Well, you know, applications almost never exist in a vacuum. You almost never have just a binary. There’s a binary, and there’s a whole bunch of shell scripts that surround it, the care and the feeding of the app.

Sam: It’s the practices of what you do with the application.

Jason: Yeah, it’s administering the back-end utilities. It’s scripts that add users to individual databases. It’s managing all the little pieces of the app. That’s in a lot of cases script based. It’s not enough to just have an application compilation environment. You have to have the runtime scripting environment as well, which is the KornShell and the awk and the grep and the curl and the sed and the other 350-whatever utilities that you need to have a real functioning shell environment that’ll make a developer or an admin happy. And that’s what gives you an environment that’s an awful lot like what you’d see on a UNIX system.

Sam: Well, I’ll tell you what, Jason, we’re going to have to probably pull you back once you’ve got that book completed, in a few more chapters, and impose on your time again.

Jason: Hey, always happy to come and talk.

Sam: Yeah, this has been fantastic.

Jason: And fun.

Sam: Thanks

Page 25 of 26


Jason: Thanks a lot.

#### End of Interview ####

Page 26 of 26

Faces from the Collective: Shared memory anyone? · Faces from the Collective: Shared memory...

Documents

Transcript of Faces from the Collective: Shared memory anyone? · Faces from the Collective: Shared memory...