Linux and Open Source

Cloud Blog on Wired site: Time To Value

2011-11-03T12:24:00.000-07:00

Well, I guess it is time to get back to blogging. I just had a collaborative blog post with Terri Virnig, our Cloud Business Executive in IBM's Systems and Technology Group. In part we try to highlight one of those key reasons that people have been looking to the Cloud: How to improve their "Time To Value". In other words, from the time that a team has a glimmer of an idea for a new project, product, or service that they would like to create, how quickly can they get that service to market.

As I pointed out in the Wired blog entry, we still see many people in industry that are finding it can take 6-9 months to get a project from concept to start of development. I was even presenting to a group recently where one participant pointed out that it often took them a year to get a new project started, in part because of hardware acquisition cycles, capital funding release, lead times on delivery, software selection, integration of software components, infrastructure integration (assignment of IP addresses often being a time consuming area there!), access to storage, backup strategy, security assessment by the IT/security organization, and the list often goes on and on.

Anyway, the focal of the article is about our recent announcement of our SmartCloud product family, including a particular project from my team known as SmartCloud Entry - a basic self-service environment for managing pools of servers, network infrastructure and storage which integrates an image (or virtual server) catalog, and hardware management in a relatively small footprint server.

I'll try to generate a few additional entries over the next few weeks which highlight some of the capabilities of SmartCloud Entry and its benefits, as well as some idea as to how it fits into the overall SmartCloud family. And, I'll try to share some of the key thinking that we are involved in with IBM Research, our clients, and our various development teams around Cloud uses, Cloud issues, and new Cloud capabilities.

Public clouds fail more visibly!

2009-05-14T09:27:00.000-07:00

With great fame and visibility also come great notoriety, at least in the case of failure. Twitter was alive this morning with the tag #gmailfail with comments like "We are receiving reports of major outages on the West Coast and East coast. Canada + UK seem unaffected so far." and "Receiving reports of total Google Services Fail. Maps, News, Apps, Reader, Gmail, all affected. #gmailfail". Some speculation placed the blame on google analytics ("Google service issues across all apps and other web pages appear to be caused by problems with google analytics #gmailfail") while others were speculating that the problem could be with portions of the internet backbone. Some felt it might be the end of the world (" feels very weird to have google and gmail down. Is this a sign of the end of the world? #gmailfail") - are we *really* that dependent on our email and search now? .

There is a summary of activity going on here as well with reports of outages via AT&T. Google notes that the problem affects a small subset of users here. Rumor is, though, you can't see that site if you can't access google. ;)

It will be interesting to see the summary of why but one thing that is clear: the level of visibility in the case of a failure increases rather dramatically. Another observation is that determing root cause of the problem is complex - isolating to a set of affected users, analyzing the connectivity between the users and their cloud based resource and internally at the cloud provider assessing the failure require very good diagnostic skills and access to the data center.

This isn't the first (or last!) cloud outage. And, perhaps the worst problem is that people were deprived of their email for a small number of hours. But perhaps it points out that the cloud isn't ready yet for all workloads.

And a late update: cnet news reports about a connection between YouTube, Google News, and the outage. (Thanks, Nish!)

Another interesting update here that was passed on via facebook. The graph here is very interesting as well with the dramatic traffic drop which implies just how much data is going on over the existing network infrastructure just for email and search driven activities.

Near Final session which covers Lightning talk readouts from some of the working groups

2009-04-10T12:43:00.001-07:00

Okay, I was slow and missed some of the updates, so this is more sketchy than I usually am, but here are some of the readouts.

gnome mobile

Maintainers of various projects were able to meet with users of those projects across the vendors. There is a lot of talk of fragmentation across the embedded space, mimo, moblin, mobile can help demonstrate that the fragmentation that exists isn't as dramatic as it may appear.

open printing

Still a shortage of manpower; it would help to have LF be more of a mentor at the google summer of code. We get 10 students for three months from the google summer of code. One will be working on getting JDK into the LSB. One or more will be working on wireless.

There are still some challenges in how openprinting and the distributions integrate the downloading of new drivers for new printers. Continued to discuss the common open printing dialog. There wasn't enough time so next year there is a proposal to start several days before the summit again to provide more time for discussion.

moblin

Open Linux targetted towards mobile technologies. Discussed what components are part of the stack. Includes connman and mojito as two topics. A couple of sessions on clutter (sp?) on using actors and timelines to create an animation in your application. A track on the moblin SDK. Last track was about porting applications to moblin. And a discussion of the changes between moblin v1 and v2. Talked about creating a moblin compliance profile for LSB with the LSB team.

HPC track

9 of the 10 largest computers in the world run Linux. Roughly half of the people who run those sites were present to meet with the Linux Foundation community.

Tracing solution

mostly non-competing, er collaborting projects started yesterday. Christoph Hellwig talked about how they worked to find some agreement on some very low level issues. Members were assigned action items, many of which worked through the night to address their action items for today.

Jim at the summary pitched the upcoming Linux Events that the Linux Foundation is sponsoring.

There are a few sessions this afternoon but the official event ends with this summary. Thanks all for reading (or attending!). And feel free to post pingbacks if you documented any of the sessions that I did not attend.

Christine Hansen on Earning the Next Generation of Linux End-Users

2009-04-10T11:55:00.001-07:00

Earning Future End-Users Now
Dr. Christine L.E.V. Hansen, CEO of Le Ciel
Arguably the largest divide in Linux remains the divide between developers and end-users. Developer oriented forms of communicating, like mailing-lists and wikae, are often the sole sources of information about projects. The future of Linux relies on connecting the developers, and the corporations, with the end-users who may not know what Linux is. What this end-user will know, or will have heard, is: 'somewhere there is free software.' Their next question/search will be: 'where can I get it?'As a point of departure, I propose a web site in plain language which serves as an index of Linux projects. All projects: commercial, nonprofit, ones in process, big, small. Obviously, voluntary participation. All together in one, search-able place. Audience: users, developers, corporators. The technical level is not as important as the pragmatic and visionary level, which ought to be high. High collaboration factor.

What the World sees:
1) Too slow
2) No 'instant ignition'
3) Product/project redundencies
4) ---oops missed it ---

Christine provided a distinction between "pure" and "applied" developers - pure being much of the first wave, idealogical, focused on Linux for themselves; applied being more focused on projects, more aware of their end users, more often speaks both tech and a human language, may be outside of North America.

Next Step: Step Up
Instant ignition, in their own language(s), solve the rediculous problem of redundant effort, introduce a multilingual Linux web presence, empower global Linii. (is that a plural for Linuxes?)

Christine pointed out that Transparent Software often has an organic life that may help it to live much longer than many of the short term trends.

Stormy Peters on Marketing Free and Open Source

2009-04-10T11:26:00.001-07:00

Marketing Free and Open Source Software
Stormy Peters, Executive Director, GNOME Foundation
Open source software solutions are now very good technically - the remaining problem is how to get the word out. Marketing in an open source world, with a volunteer marketing team, small budgets and an open source development model, is often very different than the world of traditional marketing. In addition, projects are also advocating for "free and open source software" at the same time they are marketing their solution. Come share your best open source software marketing practices and discuss them with others.

Stormy has been busy talking a lot and is low on voice, so she generated a slide per sentence.

1) most people in open source are developers not marketers

2) marketing open source is different from proprietary products since the product is "not for sale"

Typically "not for sale" means that you are not collecting money and sometimes you are actually *asking* for money. And, it tends to mean that you don't really know who your users are.

And, we still tend to need to explain what open source software (OSS) actually is. And then, OSS doesn't really matter to many users. So, for open source marketing, who is your audience?

Recruiting developers, users, distributors, partners, ... ?

That's why it is hard.

What types of marketing do we do today? Looking at gnome, we have web pages (some good, some bad) and most of the best are wikis. Events, such as GUADEC, ability to set up a table at an event. Facebook, twitter, identica, linkedin, press releases, product materials such as sponsorship materials or application summaries.

So, what else do *you* have in your community?

Are there some best practices and how can we work together?

Question from the audience - is this true for all software communities? Smaller communities really focus on growing community, but also may have dual licenses which allow proprietary distribution as well. Response from the audience: you clearly need to identify your audience and what you want to say to them. Most open source projects don't have budget to develop materials, but the Linux Foundation video contest enabled guerilla marketing and viral marketing at very low cost.

What is our message? Often we are marketing our values as opposed to selling our "products".

What are our goals? Values don't always relate to our end product. But open source projects often try to "sell" their values. Some users really *do* buy based on the values. But what percentage really purchase based on just the values?

It is about the applications for many audiences. We had some discussion of brand development, association of concepts or values with brands and what actual mental associations are drawn by non-open source users when they hear "Open Source Software" or "Linux" or "Gnome". What associations do we think people should have? "Free Software" obviously implies Free - does it imply usability, applications, ability to do your job at home, etc. Is there a way to collect input from new audiences to find out what their current associations are with those "brands"? And, is there a way to focus marketing around some of those associations that we would like to see with the higher level brands? And is there any agreement on what the higher level brands really are?

United Nations used "Transparent Technology" as opposed to Open Source and rarely is "Free Software" used in that context - which is important to understand when describing your audience.

Mozilla released a guide recently on how to do community marketing.

Lots of good discussion during the session, definitely some food for thought.

Linux Foundation Collab Summit: the Meld project for Embedded Developers

2009-04-10T10:36:00.000-07:00

Benefits to a Social Approach in Development
Joerg Bertholdt, VP Marketing, MontaVista Software, Inc. & Jeffrey Osier-Mixon, Developer Advocate and Meld Community Manager, MontaVista Software, Inc.
Community is one of the defining properties of open-source software in general and Linux specifically. To foster further innovation at the rate that the market demands, the power of community - the heart of open source - must be embraced. However, the nature of community is open sharing, which seems counter-intuitive to marketers and businesspeople working in competitive markets. There are few markets as competitive or dynamic as the embedded world. How can companies cooperate at some points in the development cycle while they maintain their differentiation at other points? How can open communities---including mailing lists, blogs, forums, corporate communities, and conferences---promote and enable this cooperation to accelerate development? This presentation draws a detailed picture of the wealth of community involvement surrounding embedded Linux and presents a look at the evolution and future of cooperative development. It also introduces Meld, a new community that enables embedded Linux developers, hardware manufacturers, and software providers to connect, share, and design commercial-ready embedded devices.

Embedded is a large market with something like 5 Billion embedded devices sold (per year?) dwarfing the server and desktop devices in the market. But the number of embedded developers contributing to the various open source community is a tiny part of the total number of contributors.

Each device in the embedded space is "similar" but never identical to all of the others. And, for each embedded device provider to be involved in all of the communities that they draw technology from is
cost and time prohibitive.

This talks core proposal is that the creation of a new community might be the answer to allowing these embedded developers to participate in the community. As a result, MontaVista helped create an open community for embedded developers called "meld" - with 1,001 members (in decimal, not binary! ;-).

Tim Bird referred to Meld as potentially the facebook of embedded developers. Jeff walked through the interactive site to demonstrate how people can connect, fill out profiles, and provide some recognition for seniority within their communities and among their interests. In general, the goal is to tap and share knowledge among embedded developers as a means of building a community around a shared general interest, despite the diversity of interests of the individual members.

Meld is intended to grow organically based on member interest and will align with hardware communities such as OMAP, PowerPC, etc, the Embedded communities, the Linux communities (via linux.com?), and Embedded Linux such as elinux.org, other wikis and communities.

Their committment is: Helping embedded Linux engineers to Connect, Share and Design collaboratively.

It is too early to tell if the community has been successful, but with 1,001 members in a month it is off to a good start. There are definitely competitors cooperating in the forum, e.g. timesys, wind river, along with the host montevista.

The site is a highly modified but originally canned solution, possibly based on or related to a .NET style framework.

Cloud Computing viewpoint from Red Monk Analyst Michael Coté

2009-04-09T11:47:00.000-07:00

The next session is an analyst viewpoint of Cloud Computing

Cloud Computing
Michael Coté, Industry Analyst, RedMonk
Linux is at the center of what's come to be called Cloud Computing. As with open source, SOA, and Web 2.0, the tech industry has quickly fallen for cloud computing. The reasons are compelling: promises of scalability, low cost, flexibility, and light weight process. The core question about cloud computing is how it effects the industry and cultural position of operating systems and other "raw infrastructure": does Linux "matter" in cloud computing? How might the Linux community evolve - or be forced to evolve - in a wave of cloud computing enthusiasm? Is cloud computing an opportunity or threat for the Linux community? Or is it just another shiny object of distraction? This talk will discuss these questions and more.

Cloud computing now is like early SOA: It's Silly-putty!

We'll take a simple definition and go with it.

How does "Linux" fit in? SWOT. Strengths, Weakness, Opportunities and Threats...

Primarily focused on Cloud in the key three *aaS's (pronounce that carefully!): SaaS, PaaS, and IaaS.

Why Cloud Copmuting:

Users: Cost, Flexibility, Elasticity/Scalability
Vendors: new business models, new features, lower cost of ongoing maintenance (?)

Cloud Computing has sort of supplanted discussions around autonomic or some of the *aaS terminology.

Hype-fed, semantic confusion - public vs. private

Retraining - multi-thread development, dynamic operations

Ending up paying more, e.g. $20k on-premisies vs. $150k off-premises.

Lock-in - the current public cloud solutions debateably have lock-in today. There are some activities to try to use standard/de-facto interfaces like Amazon's EC2 and Eucalyptus, Rightscale abstracting or layering on top of an infrastructure, etc.

Legacy concrete - moving workloads to the cloud are not as simple as a finger snap. Often need to replicate a cloud solution, maintain internal and external solutions, and pay the additional cost for having two solutions running concurrently while evaluating and moving to public clouds.

Virtualization turns out to be more important - "fog computing". Maybe Virtualization is more important and will outlive the Cloud Computing hype today.

When looking at pubic cloud versus an internal deployment, you still have to look at the total cost of the solution. This includes capital versus operational expenses, management expenses, ongoing expenses, etc.

IaaS: Amazon EC2, S3, etc.

PaaS - force.com, Microsfot Azure

Sun Cloud, 3Tera Rackspace, & reborn hosters

Automation and provisioning people helps enable all cloud-like solutions, such as Puppet.

Linux Strengths
- Appealing for it's (potential) cost of zero
- Reliability, known quantity "at scale", breadth
- Malleable & Transparent
- Easier for IT management and tooling
- Virtualization
- Existing, Linux compatible applications

Linux Weaknesses
- Focus on the OS level, not applications and usage
- Weak connections to the development platforms, with exceptions like LAMP
- Virtualization and Automation Fragmentation
- Susceptible to "bad citizens"

Linux Opportunities
- OS for Cloud client - Netbooks, RIAs, "mobile"
- Model driven automation
- Fragmented air-space - everyone wants a cloud, everyone wants Linux
- Easy way to boot-strap into using Linux - deploying Linux has become easier but with the cloud, creating new Linux images is downright trivial.

Linux Threats
- SaaS and PaaS means "the OS doesn't matter" - cars with hoods that don't open (past: VM & .NET CLR)
- Commercial vendors creating new closed source worlds, e.g. Apple
- Cloud Distro Madness - QA & Support matrices
- Cloud consolidation & collapse - eggs in a basket

Conclusions
- Worst case scenario: Cloud Computing is nothing and Nothing Changes for Linux
- Best case scenario: Many, many more Linux instances
-- In short, the best of all worlds, can effectively do nothing from the point of view of the Linux community and either reap the benefit for free or let the wave of hype pass by without impact.
- Thinking ahead: transitioning "brown field" applications. Possibly more interesting to figure out how to move existing, older applications into the Cloud environment. How do you get most of that old, boring, Enterprise software out into the Cloud?

So is there any thinking about interoperability with other Clouds? And how does this interact with the view of the Gartner Hype Cycle - will we cross the chasm here because of inhibitors like cross-cloud interoperability?

Cloud computing is mostly going to be just another "new" option that will co-exist along with all of the other options.

Question on internationalization: some countries may be very specific to specific language subsets within a particular country. Then, how do you benefit from others that are localized to a specific language? How do changes fold back into the open source communities?

Privacy and data security is going to be a huge part of this. Does this problem still exist when moving the cloud "in-house"? Again, various departments will be sharing the same infrastructure, in theory. Aren't the same privacy constraints a concern? And yes, to some extent, although employment guidelines help provide some protection. Amazon recently put out a short white-paper on how to comply with HIPAA requirements and some description about how Amazon does not have access to your data. But, oh, btw, engage your own laywer: this is not legal advice. The privacy space is likely to be complex for a while.

Will virtualization and cloud computing reduce the need for network and system administrators in the future? Sort of like the question that asks if we put robots on the assembly line will we remove people or reassign them? General answer right now: the complexity changes and the costs over time may go down but don't expect any immediate reduction in the need for systems administrators. But of course, learning new skills is always necessary in our fast moving, high tech environment.

KVM update by Chris Wright

2009-04-09T10:53:00.000-07:00

At a glance - KVM is a Linux kernel module that turns Linux into a Hypervisor.
Requires hardware virtualization extensions - uses paravirtualization where it makes sense
Supports x86 32 & 64 bit, s390, PowerPC, ia64.
It has a competitive performance and feature set
Advanced memory management
tightly integrated into Linux

A hypervisor needs:
- A scheduler and memory management
- An I/O stack
- Device Drivers
- A management stack
- Networking
- Platform Support code

Linux has world-class support for this so why reinvent the wheel?
Reuse Linux code as much as posssible.
Focus on virtualization, leave other things to respective developers.
Benefit from semi-related advances in Linux.

KVM features
- buzzword compliant - VT-x/AMD-V, EPT/NPT, VT-d/IOMMU
- CPU and memory overcommit
- Higher performance paravirtual I/O
- Hotplug (cpu, block, nic)
- SMP guests
- Live Migration
- Power Management
- NUMA
- PCI device assignment and SR-IOV
- Page Sharing
- SPICE
- KVM autotest

One of they key points is that many of the capabilities come directly from the underlying Linux kernel providing these features.

Libvirt Features
- Hypervisor agnostic: Xen, KVM, QEMU, LXC, UML, OpenVZ
- Provisioning, lifecycle management
- Storage: IDE/SCSI/LLVM/FC/Multipath/NPIV/NFS
- Netowrking: Bridging, bonding, vlans, etc.
- Secure remote management: TLS Kerberose
- Many common language bindings: python, perl, ruby, ocaml, c#, java
- CIM provider
- AMQP agent - High bandwidth, bus based, messaging protocol to enable the ability to manage very large numbers of machines. Common with Wall Street Linux customers.

oVirt features
- Scalable data center virtualization management for server and desktop
- Small footprint virtualization hosting platform
- Web UI for centralized remote management
- Directory integration
- Hierarchical resource pools
- Statistcis gathering
- Provisioning, SLA, load balancing
- Currently built on top of KVM (not a hard requirement)
- Currently directly built on top of Fedora, but again not a hard requirement
- oVirt is about managing the hardware resources as well as the guests, includes ability to include agents on the guests and monitor the guests that way as well.

Question: can you use oVirt to manage guests on Amazon's EC2? In principle, yes, but there would be a bit of work to enable that, mostly because it includes the hardware provisioning access. In some sense, oVirt because the "owner" of the physical machine to enable virtual machine deployment. It would depend on libvirt running on the physical nodes within the "cloud", e.g. Amazon's EC2.

Newbie question (yes KVM is also Keyboard/Mouse/Monitor multiplexor, but not in our session today ;-): How does KVM compare to OpenVZ and Xen. OpenVZ is focused on containers running on a single Linux instance. The guests are not as isolated and do not include a complete and unique kernel & OS instance as is done with Xen or KVM. OpenVZ is more like chroot on steroids. Again, more like VMware's ESX. Xen is basically a microkernel approach to a hypervisor where KVM is a "macro kernel" approach. Xen allows modifications to a virtual machine to run as a paravirtualized system for performance considerations. KVM can run a .vmdk image but it is probably more useful to convert to a KVM-friendly format. Paravirtualizing I/O reduces the enormous number of traps that are otherwise present in a fully virtualized hardware environment where the hardware does not have full IO virtualization or an IO MMU.

<at this point we hit the break but Chris will go into his next 40 slides for those that want to stay ;) >

KVM Execution Model
- Three mdodes for thread executiion instead of the traditional two
. User mode
. Kernel mode
. Guest mode
- A virtual CPU is implemented using a Linux thread
- The Linux scheduler is responsile for scheduing a virtual cpu, as it is a normal thread.

Guest code executes natively apart from trap'n'emulate instructions
- performance critical or security critical operations are handled in kernel, such as mode transition or shadow MMU.
- I/O emulationand management handled in user space such as qemu derived code base and other users are welcome.

Large page allocations are currently pinned and never swapped out. This is a current downside to using large pages within KVM.

KVM Memory Model
- Guest physical memory is just a chunk of host virtual memory os it can be
- swapped, shared, backed by large pages, backed by a disk file, COW'ed, NUMA Aware
- The rest of host virtual memory is free for use by the VMM, low bandwidth device utilization

Linux integration
- Preemption (and voluntary sleep) hooks: preempt notifiers
- Swapping and other virtual memory management: mmu notifiers
- Also uses the normal linux development model including small code fragment, community review, fully open

MMU Notifiers
- Linux doesn't know about hte KVM MMU
- So it can't: flush shadow page table entries when it swaps out a page (or migrates it, etc.); Query the pte accessed bit when it determines the recency of a page
- Solution: add a notifier for 1) tlb flushes 2) access/dirty bit checks
- With MMU notifiers, the KVM shadow MMU follows changes to the Linux view of the process memory map.
- Without this, a guest would be able to touch all user pages and the base Linux wouldn't know that those pages could be swapped out.

Paravirtualization
- Not nearly as critical for CPU/MMU now with hardware assistance; highly intrusive
- KVM has modular paravirt support: turn on and off as needed by hardware
- Supported areas: 1) hypercall-based, batched MMU operations 2) Clock 3) I/O path (virtio) [The last is the most critical currently]
Now native shadow page table operation is generally more efficient than paravirtualization so paravirt is rarely used with KVM today.

Virtio is cool
- Most devices emulated in userspace with fairly low performance
- paravirtualized IO is the traditional way to accelerate I/O
- Virtio is a framework and set of drivers:
- A hypervisor independent, domain-independent, bus-independent protocal for transferring buffers
- A binding layer for attaching virtio to a bus (e.g. pci)
- Domain specific guest drivers (networking, storage, etc.)
- Hypervisor specific host support

There is a tradeoff between moving driver support into kernel vs user level. user level provides better security isolation and often negligble performance degradation. Using dbus for communication is typically about 60 milliseconds (did he really say ms ?? or microseconds? ) either way, it introduces some latency. Plan to move the virtio drivers back into kernel to measure and see if there is any noticeable difference.

Infiniband and ofed (?) do they work? IB just works, driver functions, would you have RDMA support right to the target. Answer is: It depends. Certainly possible but it gets pretty complicated. Without assigning an adapter to the guest it becomes difficult to register a set of pages with the driver for direct DMA.

Xenner is a mode you can run QEMU in. An independent application that uses KVM.
Emulates the Xen hypervisor ABI - Much, much smaller than Xen
. Used to run unmodified Xen guests on KVM
Has been going on for quite a while and will soon be a part of QEMU directly.

Roadmap:
- QEMU improvements and integration: libmonitor, machine description
- qxl/SPICE integratoin
- scalability work: qemu & kvm
- performance work
- block: i/o using linux aio
- Network: GRO, multiqueue virtio, latency reduction, zero copy
- Enlightenment - the ability to receive calls from Windows guests. Hyper-V requires VT.

http://linux-kvm.org, http://libvirt.org, http://ovirt.org.

Main contributors: AMD, IBM, Intel, Red Hat
Typical open source project: mailing lists, IRC
More contributions welcome.

Ksplice: reducing the need for kernel reboots

2009-04-09T09:59:00.001-07:00

Ah, good morning... This morning I'm moderating a session on New Technologies - the first presenter is Jeff Arnold who is covering Ksplice. From the LF Program:

Ksplice

Jeff Arnold, Lead Developer, Ksplice
Today, when Linux users apply security updates to their systems, they are commonly informed that they must reboot in order to finish applying the updates. Since rebooting is disruptive, many users and system administrators delay performing these updates, despite the increased security risk--more than 90% of attacks exploit known vulnerabilities. New technology out of MIT, called Ksplice, enables running Linux systems to stay secure without the disruption of rebooting. This talk will describe the core innovation behind Ksplice and how this technology will improve the security and maintainability of Linux systems.

As good as Linux is (transcendent? ;-) it still has bugs. Some of these include security problems which typically need updates as quickly as possible. However, people do not enjoy the scheduled downtimes, especially in this world of round-the-clock access to services. Reboots also have the problem of losing software state such as network state, application state, etc. 90% of attacks exploit known vulnerabilities, so getting patches installed more quickly would reduce the level of vulnerability. Also, applying patches sooner avoids the need to wait for a scheduled maintenance window. Ksplice currently adds negligible performance impact. If any impact is found, please notify Jeff asap!

Ksplice works on any release since Linux 2.6.8. he initial release is in April 2008 (GPLv2). Ksplice has tools today in Debian sid, Ubuntu Jaundy, Fedora 8-10. It is currently proposed for mainline and has 5 engineers working on Ksplice full time.

Question from the audience - doesn't virtualization obviate the need for this type of technology? Answer: Not really. Even the unlderlying OS supporting, say, KVM, still needs to be patched. The impact of downtime or mandatory migration for all virtual machines running on that physical machine is still extensive. And, taking down the network state or application state for a virtual machine still impacts the workload.

Jeff provides an example of code that needs to be updated, and shows a way to apply a patch and generate a ksplice patch. He then shows a binary exploit, followed by a complex patch including asm code and C code to a couple of files. Then uses Ksplice to build two kernels, one with and one without patch. Then with a simple sudo ksplice-apply applies the kernel patch, ksplice-view shows what ksplice updates have been applied, and a re-run of the exploit code fails to provide him with root privileges. ksplice-undo allows the patch to be removed.

These changes only update the running kernel. However there are also tools which update the kernel for the next reboot or allow a set of rc scripts to apply updates at next boot to the same kernel.

Source for the kernel must be available to apply the patch. If you try to build and install against the wrong source version Ksplice will notice if the kernel changes. You must also use the same compiler and assembler version. Could a vendor provide enough information so that an end-user could create these patches without the full source? Possibly yes. Or could a vendor build these kernel modules for distribution to end users to allow the end users who don't have full kernel source easily available? Yes.

Jeff showed a Ksplice Uptrack Manager much like the Ubuntu software update menu. Looks like like a normal package manager but with patches getting spliced into the kernel? The kernel stops for about 700 microseconds (0.7 milliseconds) during this update.

Question from the audience on what the likely model is here for distributions. Does every vendor need an infrastructure for their patches. Ksplice is hoping to be able to create these updates for a number of distributions. A distro vendor member would like to have both the mechanism for updating any set of pre-build kernels as well as the running kernel.

Next Jeff talks about the mechanism for changing the code based on conversion from a source code patch to the binary updates.

Identify which functions are modified by the source code patch and then generate a "replacement function" for ever to-be-replaced function and then start redirecting execution to the replacement functions.

pre-post differencing. Pre source is unmodified, post-source is the post-patch application source code.

Ksplice compiles both pre and post, then compares the two binary object files and extracts the replacement functions.

Shows a picture of foo() and foo()' (foo() prime), adds a jmp instruction at the beginning of foo() to the beginning of foo()', return from foo()' returns to caller of foo().

There are some symbolic references that also need to be patched as well.

Need to make sure that foo() and foo()' are not running at the same time. Temporarily grabs all CPUs, makes sure it is not running one of those functions, if necessary, abort(rare).

Design only allows changes to code, not data structures today. ksplice_pre_apply() and ksplice_post_apply().

In some cases, a patch may need to iterate over all CPUs or use the ksplice_apply() call directly to apply the patch.

Hypothesis: most linux security patches can be hot-applied with Ksplice. Used all security patches from 2005-2008, and they were able to hot-apply with 88% without change, and 100% with only about 17 lines of code per patch of additional code.

Debugging with Ksplice can do some of what kgdb and SystemTap can do because it does live patching. Can insert code almost anywhere and discover any symbol value. x86_32, x86_64 and ARM.

Jonathan asks how we avoid making the problem worse by creating new versions of a kernel that was never really seen by a distro or a kernel developer? Is the concern that there is unreviewed code? Or is it that there are a larger variety of running kernels modified "in place". You could use Ksplice for just the 88% of directly applied changes without having some of the extra code (which could itself have bugs) and limit some of the risk. Any new code for the remaining 12% is intended to be as absolutely small as possible to minimize the chance of introducing new bugs.

Is this a portable root kit? Well, sort of if you have root already and can load modules, yes. Does it make it a little easier to generate a module which can be loaded to root the machine? Probably. But black hats already have these techniques and it gives the white hats an easier way to keep up with the black hats. It is possible to also use cryptographically signed modules with Ksplice as well to improve security overall. There are also a lot of tools to see what ksplice updates are installed, sets taint flags, etc. to make sure that people can tell what was changed.

This should work well with the stable trees as well although they've had to remove some patches to make a stable tree do a live update to a running base kernel.

So who won the Linux Foundation Video Campaign?

2009-04-08T17:21:00.000-07:00

Jim Zemlin: "So, the videos that won this campaign were all great..." The contest outstripped the corporate entries for a typical contest of this type, with four to five times the number of submissions as compared to comparable corporate ad campaigns.

Here are the winners: http://video.linux-foundation.org/contest/winners

My personal favorite is actually the first runner up - it is fun. ;)

Why can't we All Just Get Along? Microsoft, Sun, Linux reps discuss potential for interoperability

2009-04-08T17:14:00.001-07:00

Roundtable Discussion: Why Can't We All Just Get Along (View Summary)
Jim Zemlin, Executive Director, Linux Foundation
Ian Murdock, Vice President of Developer and Community, Sun Microsystems
Sam Ramji, Sr. Director, Platform Strategy, Microsoft Corporation

Microsoft: What would you have done differently? Sam Ramji: I would have on Day #1 built a relationship with the Legal team. Engineers adapt quickly but Lawyers tend to take a little longer to be educated on the trends and directions and to understand the ramifications. The exact quote might have been: "Engineers iterate and get to the right solution, Lawyers mitigate risk..."

Sun: Open Solaris work from Ian - he entered a little naively due to the inherent inertia when working with a large company. While building up a consensus finally happened, it took much longer than expected. Once the inertia is moving in the right direction, it is much easier to get things done.

Jim to Sam: It is clear that MS sees the computing landscape changing. Is there anything that you'd like to see from this crowd, beyond just going away and leaving us to have the dominant market share?

Sam: View Sam as the "unelected rep within Microsoft" to help work to advocate for changes in behavior within Microsoft. Have built up Linux on top of Hyper-V as well as Windows on top of Xen within MS. Hopefully the Linux community can provide positive reinforcement for the things done well and not just kick the MS for the things done wrong.

Jim: Why does MS care about the open source community? There is often an assumption that MS has a nefarious purpose behind their actions and the open source community is trying to figure it out.

Sam: Two answer: First, he'd like to see computing just get better. Sam sees that greater efficiency in computing drives greater productivity within the economy.

$60 billion in revenue - what is the next engine for growth? The better MS works with other products the greater the growth potential for MS products. A lack of interoperability will likely be an inhibitor to revenue growth.

Jim to Ian (via Twitter): Who is the largest contributor to open source in the world: Ian: Sun! The entire company is built around open source, literally. Working with other companies to ensure that their products interoperate. Jim: What is Sun going to do with mySQL? In Cloud computing or Web 2.0 companies, whatever you want to call them, open source is a major underlying technology which is a place where mySQL fits in nicely.

One example of agreement between Linux, Sun and MS is on accessibility standards for people with disabilities. Standardization in this space is a place where we could all work together.

Ian: Sun did not foresee the rise of Linux and did not get the chance to incorporate it into their strategy until too late. (Contrast this with Oracle's insight as viewed in an earlier posting with hindsight).

Jim: is there any possibility of ZFS and dtrace under a GPLv2 license? Ian (after some hesitation) "There is a possibility, yes."

Unfortunately I didn't catch all of the debate in this session because the dynamic was very interesting. The fact that this panel exists is a good sign that there may be some collaboration among some more extreme competitors. Of course, this is a major aspect of this part of the industry in which many competitors seek to find common ground for collaboration, advancing their common interests.

All in all, kudos to Ian, Sam and Jim for setting this up and running with it.

LF Collab Summit: Linux in the Enterprise: The Journey, Milestones and What's Ahead

2009-04-08T15:47:00.000-07:00

Edward Screven, Chief Corporate Architect, Oracle
Date: Wednesday, April 8th
Time: 15:15 - 16:15
Location: Imperial
Summary: Linux has come a long way from its modest roots in the early 1990s to now being the operating system of choice for data center deployments. Join Oracle's Chief Corporate Architect, Edward Screven, as he discusses the importance of Linux in the industry, and how Oracle views and supports Linux. Edward will also highlight his thoughts on advancements needed and being made in Linux that will continue to keep it on the top.

Why Does Oracle Care About Linux?

Oracle wanted customers to try to standardize on shared pools of low-cost servers and storage as a way to save money on hardware, break down the existing Silos within the data center, and simplify the growing heterogeneity of Data Centers. Windows was not a realistic option, BSD was considered but Linux appeared to have a better chance, although in hindsight, this relatively pivotal decision was as much luck as it was a good choice.

In 2002, Larry Ellison said "We will run our whole business on Linux". Today that statement is true.

Chuck Rozwat said "We will run our base development on Linux for all of our products" Today they do.

Oracle Enterprise Linux is primarily a support business exactly tracking Red Hat Linux.

$954 million in 2006 to 1.2 billion in 2007 (Gartner) makes Linux servers the fastest growing OS sub segment, growing 25.6% over that time period.

Oracle claims that they may be the world's largest user of Linux, including their Traditional IT, Development, Oracle On Demand, and Oracle University. 84,000 servers, 10 PB storage, all running Linux. 1000 dual CPU machines in "The Farm" - their test grid. 75% running Linux, 2000 jobs run simultaneously.

What helps linux succed in the data center?

- Cost effective - total cost and marginal cost

- Standards Based, extensible, 3rd party support, open

- Enterprise Ready - scalable, reliable, manageable (at the individual box level)

- Tested configuration ready to deploy

- Enterprise-Class Integrated Support

Oracle strongly believe that Virtualizatoin Changes the Game - turns pools of servers into a fungible resource. Oracle VM is a product in this space, based on Xen, supports Linux & Windows.

Used in Oracle University, 1/6th the hardware, CPU utilization increased from 7% to 73%, Servers to administrator ratio increased 10X, Revenue per server increased 5X.

In Oracle on Demand, every customer required at least 6 machines, test, development, multiple tiers for database and servers, etc. With virtualization cuts that number down to half that or better.

Next step: Make Linux into the Standard OS for the Data Center. He believes Windows will always be the default desktop OS, but the standard Enterprise/Data Center OS should be Linux.

Endorses the btrfs file system

- General purpose file system

- Handle large storage

- designed for repair and reliability

- copy on write with efficient snapshotting and checksumming

- Manage multiple devices under the file system in raid striped and mirroed configuration

- http://btrfs.wiki.kernel.org

- In kernel 2.6.29

Comments on SystemTap as not being open from the very beginning which has led us to where we are today.

The Opportunity for Linux in a New Economy

2009-04-08T14:52:00.001-07:00

Al Gillen is back for the second year at the Collaboration Summit.

Outline:

Economic Impact
Trends from past recessions and how that applies today
The role of virtualization software
Outlook for the Linux ecosystem
Essential Guidance

There is a slowdown in IT spend - a Capital Expense (capex) reduction is mandatory for many customers, work in North America, second is Western Europe, less in Asia Pacific, at least for now. Nonpaid solutions are likely to be hot: Linux, open source DBMS, middleware, tools. Watch for update on non-paid virtualization solutions.
ROI Window compressed dramatically, paybacks not realized this fiscal year are nonstarters. Difficult to justify new initiatives today.
New migratoin initiatives are unlikely. If a migration was not already underway, it is unlikely to start now. Existing skills will determine whaat is and is not done.
Everybody wants to declare victory in a down market.

In a down economy, everyone is loosing money. No business does well, but we look more at what dynamics will change as a result of the current downturn over the next three years.

A chart on x86 server shipments shows a marked drop in Q4 2008 and is projected to return to current previous highs around the end of 2010.

Roughly a drop from 8+ million servers a year to about 7 million x86 servers a year.

It appears that the software spend trend matches the hardware trend, although the software trend remained positive (ranging from a high of 14% growth per year to a low of about 2% growth per year) with a projected high of about 7% growth per year by 2013.

Legacy of the 2001-2002 recession: That was the original demarcation line for the acceptance of Linux in the market place. Customers also began to buy major new servers for Linux deployments around that time. This also drove the build for the software on Linux. About that time CIO's claimed that there was no Linux in their data center while people in the trenches pointed out that they were using Linux at the fringes and the grass roots entrenchment had begun in earnest. That also drove the beginnings of the standardization around Linux. Linux was just then available on a number of diverse hardware platforms, driving a level of commonality in application availability - especially open source applications such as Apache, across the Enterprise.

Virtually all of the paid market for Linux is in the Enterprise space today. Some geographies are focusing on the non-paid distros.

After the recession, there was a strong growth period for several years but also that recession drove a strong push towards virtualization and consolidation.

"Free" servers became widely available thanks to virtualization software. Unix is increasingly under siege from Linux... and Windows.

Cloud might get a boost but revenue from cloud Linux might not.

Windows, of course, does not go away. Nor does Microsoft cease to be a fierce competitor.

Tranitions eventually mandate rationalization and assimilation - Managing nonpaid OSes, Managing hypervisors and guest OSes.

Virtualization as a Solution

Lifecycles for OSes and apps will be extended. Solutions that foster cost avoidance will be favored. OSes that were in use will probably stay in operation much longer than they have historically. For instance, RHEL4 apps and environments may remain in the data center for more than the typical 10-15 years and more like "as long as the application has value" - which could be over 20 years in some cases.

Solutions that cost $0 will be tested and adopted. The initiatives will result in permanent changes to how IT does business.

The Virtualization Effect: We see a decoupling of hardware and software, via virtualization. Virtualization takes two basic forms: independent guests ("stand alone" OS) - associated with nonpaid copies, less c; affinity with less critical workloads.
Replica guests ("child" copies) - Associated with enterprise distros, more critical workloads, virtualization rates higher with enterprise subscription. The rate of virtual systems to physical systems is likely to increase to 2-1 in the datacenter in the next 18 months - where it was 1-1 up until about three (?) years ago.

Where does virtualization go next?

Pricing has been driven to $0
Value add moves to management
Integration with hardware, OS
Uniguest installations
Heterogeneous hypervisor management
Managing offline images critical

Al predicts that it might take 3-5 years for KVM to work its way into datacenters as a primary virtualization platform. What this means to Xen is unclear but probably does not impact most enterprises much over the next 3 years and they will have time to devise a migration strategy.

Linux and Cloud Computing

Infrastructure Cloud - services such as CPU, networking and stroage; often presented as a virtual machine over the Web; user installs and manages their own OS and applications; pay by the megabyte, gigabits, MIPS, etc.; Examples: Amazon EC2.

Platform Clouds: An operating system and possibly infrastructure softare; hosted in a web-Accessible location; may proivde apploication developre and runtime envrionement: Example Any Web hosting provider

Applicatoin clouds: Virtualize and entire application (aka SaaS) Consumed as a solution or indvidual services through APIs. Examples: Salesforce.com, Google

Linux has already had a big play in all of these areas.

Outlook for the Linux ecosystem

Chart showing Linux distro sales from Red Hat and Novell - both showed positive growth without much impact up to Q4 2008 share numbers. Interestingly Red Hat was closer to $140 million in licenses where Novell was closer to $20 (from memory) - but Novell's growth was less impacted than Red Hat's, and both had nearly negligble impacts from the economy thus far.

Still forecasting a 15.7% CAGR for Linux, with a shift in what markets moving up the stack are growing. The distro itself is a tiny portion of the overall spend, hardare is third smallest, with App development and application software being the two areas that grow the post, with Services being perhaps the second largest area of growth (I can't see the exact numbers but the slides should be published on the LF web site).

Al pointed out in one of the graphs that a smaller market share easily mangifies a CAGR - so comparing Windows CAGR on a larger market with a larger CAGR for Linux in a smaller market does not mean that Linux has taken over Windows. At least not yet. ;)

In key workloads, Collaboriative, Application Development, IT Infrastructure, Web infrastructure, decision support are called out in some detail and their corresponding growth rates (worth reading the chart).

Drivers for Linux:
Capex concerns, virtualizatoin -
guest acquisition costs, ease of deployment, Non-paid Linux, larger ecosystem is good, open source layered SW
Increasing integration
software appliances
- new form factors
- new GTM sscenarios

Challenges for Linux
- Capturing new customers - particularly in a down economy, freeing up resources, staff training
- Non-paid Linux and OSS - generates on revenue, dries up survival funding
- Microsoft - Windows solutions, applications
- Generating revenue from Cloud

Essential Guidance
- Look at the current economic downturn as opportunity
- user virtualization as an integration point for commercial and nonpaid Linux
- Remember that revenue is not hte only metric that matters
- However, revenue is important
- Missed a bullet or two, sigh. My fingers are tired. ;)

Question from Matt Domsch: Is there a distinction between Public and Private clouds? Al: Yes. There are places for internal clouds and probably a hybrid between the two at many customers.

What about the guests in virtualization. Is there any preference for Linux or Windows? Al: Currently there is a larger set of Windows installations often driven by software applications. Aging Linux apps are often infrastructure related and it is often easier to replace infrastructure servers than to replace custom Windows apps in the Enterprise.

Linux rate may be similar to the windows rate in the next 6-8 years.

How do you measure paid vs non-paid Linux? It seems to some that there are more non-paid installations than the 50-50 split on the charts might indicate. Al: Based on world-wide surveys, returns say somewhere from 35-55% are non-paid. That number probably reflects Corporate users. There are a fixed number of servers out there so that number is used as a way to validate the response to some level.

A comment from the audience suggested that there might be a business tradeoff to consider on non-paid OSes - if you are paying for a kernel engineer or administrator to support the non-paid OS have you really saved money by using a non-paid OS?

Jim Zemlin wound up the session with the observation that non-paid Linux is obviously a key question. But it is worth noting that if you use a non-paid OS you won't be going to jail, unlike using some other OSes without paying.

Linux Foundation: Panel: Measuring Community Contributions

2009-04-08T13:46:00.001-07:00

Luckily I found a power outlet, so here comes more blog fodder!

Panel: Measuring Community Contributions (View Summary)
Joe Brockmeier, Community Manager, OpenSUSE

Panelists:
Jono Bacon, Community Manager, Ubuntu
James Bottomley, Linux SCSI Subsystem Maintainer
Dan Frye, VP, Open Systems Development, IBM
Karsten Wade, Fedora Project, Red Hat

Joe asked each of the panelists to introduce themselves - Karsten as a gardener, Jono a member of a metal band (did I hear that right), Dan Frye - VP of Linux Development at IBM, and James Bottomley - now at Novell and still SCSI subsystem maintainer and Chair of the Linux Foundation's Technical Advisory Board.

James: Contributions to the upstream kernel is a major source of community contributions. Karsten points out that users bear the brunt of the poor contributions. Jono points out that there are key contributions beyond just code. As an example, ten years ago the documentation project required that you know LaTeX. Today, contributions such as documentation, translation etc. are valid contributions. Joe asks what about companies that contribute market or legal review - general agreement that that is a useful contribution. Dan Frye points out that it was best said by <insert name here> in the Halloween note: People scratch their own itch. A real community allows people to contribute as benefits the contributor, as opposed to only contributing from a laundry list of features needed by the recipient. This is key to the success of a real community.

If your itch involves contributing in the Peer to Patent project, that adds value to you and thus to the community that you are a part of. James points out that code contributions are critical but so are testing, debugging, bug fixes, etc. And, the community accepts money (via the Linux Foundation) to support various Linux related activities, such as the Linux Kernel Summit or requested capabilities.

Dan: I don't see any reason to "grade" people's contributions. But we do need to remain inclusive allowing people to contribute in ways that helps to scratch their own itch. There's no need to guilt people into contributing in ways that they don't happen to find worthwhile. However, it is key to make sure that the community remains open to people so that they can scratch their own itch, to be a part of the community and to grow within the community.

Jono points that we are all still learning about the different ways that people can contribute. For instance, some people are predisposed towards certain strengths, such as python programmers or documentation writers or graphics artists. From a distro perspective, the knowledge of how new members could contribute based on their skills is a new and emerging competency.

How much thought goes into a vendor's training and education about how to make community contributions. Tongue in cheek from Joe: IBM plans everything, right? Dan laughs - we try. IBM does evaluates a new community, understanding what roles we might be able to take on, learned about styles, governance models, and try very hard to learn how to be real and effective members of the community. James points out that we also learn by mistakes. Dan points out that there isn't a mistake that IBM hasn't made.

James points out that IBM long ago attempted to make contributions in whole cloth quite unsuccessfully. IBM then went out to break down the contribution into small components which benefited the community and over time achieved IBM's own ends. Dan agreed that was a lesson IBM had to learn many times.

Karsten's perspective was that he was impressed with IBM's due diligence. His feeling is that Red Hat's approach has been more evolutionary in terms of interacting with communities and probably less formalized. James pointed out that this is a key different between an industrialized approach of "I pay you for X feature" versus the community approach were people get together to share goals and possible solutions and work collaboratively to create a solution.

James points out that the panel has collectively a lot of experience in training an organization to engage with open source and is hoping that this knowledge can be transitioned to new folks and hopefully in part to the Linux Foundation.

Andy Wilson from Intel points out that development seems to be US centric. Dan points out that IBM has a large number of contributors in India, Brasil, some in China. Joe asks for clarification - are these people at large in these countries or are they IBM employees? Dan says these are IBM employees. Andy agrees that IBM has a supportive corporate culture here, but suggests that the problem is more endemic to the Linux community as a whole. James points out that IBM's involvement has helped generate a lot more interest in developers in India becoming more interested in Linux. Dan also agreed that we are (as a community) short on contributors from eastern Europe and other countries as well as a number of other major developing countries - not just Asia.

James points out that there is also an infrastructure problem in these countries which has to be addressed as one of the key inhibitors to the growth of Linux developers in those countries.

A member of the audience asked what objective measures are actually being used - based on the title of this panel. Karsten points out there there are some objective measure such as mailing list traffic and analyzes the source of key contributors (similar to Greg KH's analysis of kernel contributors). The goal is to be able to help track the health of any community. Another measure might be IRC traffic on a project channel. James points out the work of the Greg KH and Jonathan on the git logs of the kernel to permanently track all contributions. Karten points out that community is primarily based on communication. There are some collaboration techniques such as bug work flow that provide indicators of community contribution or community vitality.

A woman from Google asked if there was a measure of success of mentoring and how mentoring improved or led to the viability of a community. James pointed out that they do some of this for the Linux kernel via the Linux kernel summit. Dan pointed out that companies don't measure code contributions as much sa effectiveness. In IBM's view, if participation in a community led to other people contributing code to address IBM's customer needs. That, in fact, was more important than measure patches which may have no bearing on value. James pointed out that no real kernel maintainer appears to have a manager which measures based on that level of effectiveness. Jono points out that measuring for the point of measuring is of course pointless. James pointed out that many people are motivated simply to get a patch into the kernel.

And with that, the time expired. Jim Zemlin wrapped up the session by pointing out that the Linux community is an excellent example of the success of a development community. And, introduced Al Gillen from IDC who will provide some measures of the success of Linux in the market place.

Kernel Developer Round Table at LF Collab Summit

2009-04-08T11:56:00.001-07:00

Panel: The Linux Kernel: What's Next
Moderator: Jonathan Corbet, Editor at LWN.net

Panelists:
Greg Kroah-Hartman, USB & PCI Subsystem Maintainer
Andrew Morton, Lead Kernel Developer & -mm tree Maintainer
Keith Packard, X.org Project Lead
Ted Ts'o, Chief Technology Officer, Linux Foundation

2.6.30 merge window just closed. Linux noted that about a third
of the code that went in was "crap". A lot of code went into the
staging tree. So, for Greg KH, what is the staging tree really
for?

Greg KH: the staging tree came out of the driver project which
provides a collection point for random drivers, including bad
API usage, bad code, rather crappy code. GregKH is now the
"crap" maintainer, er, ah, staging tree maintainer. So, about
130 drivers were merged into the staging tree, all experimental
code, mostly from drivers that have been out of kernel since
the 2.0 days. Slowly that code is getting cleaned up now that
it is consolidated and being evolved to the point where it can
be merged into mainline.

Some distributed filesystem work, aka Ceph, went in through the
staging tree: Why? GregKH: because the maintainer asked that
it go in through there.

For Keith: what are the current graphics things that Keith is
working on. Drivers used to be done all in user mode but have been
re-educated or have come to the awareness that a number of changes
really need to be in the kernel to support graphics. A number of
new APIs for accelleration, video mode configuration, and memory
management code is now in the kernel and can be used by the X11
graphics drivers. Or rather, the X11 graphics drivers provide just
one of many graphics drivers based on the in-kernel support. This
makes the graphics capabilities more accessible to graphics driver
writers. There are still some problems in the 2.6.29 code base and
2.6.30 is getting better. But most of this stuff is pretty bleeding
edge and probably should have been run through staging.

Graphics are now at a much better level of support in Linux than they
have ever been. The number of supported chipsets is finally increasing
from just Intel chipsets to include a number of the ATI chipsets
more fully supported out of the box. ATI has probably put less
developer dollars into improving the drivers as compared to Intel,
but they are getting a fair bit of help from the community and
are communicating well with the developers. Fedora 11 has shifted
to the nouveau driver for nVidea hardware which in some cases exceeds
the capabilities of the native, binary drivers provided by nVidia.
nVidia is still not working at all well with the Linux community.

The graphics community could still use additional developers and
improved vendor support in general.

Jonathan: Is there anything we can do to make the community more open
and accessible to new developers? Keith: the wayland (sp?) project
is a new windows system (not X11 based) whic his using the new kernel
APIs which would not have been possible without accelleration and basic
configuration support in the kernel. The same is true for the Cairo
project. These new APIs and kernel support should enable an increase
in the velocity of change.

Jonathan: where are filesystems going? Ext4 was just pronounced
"stable". Ted: Two community distros (fedora and ubuntu) will be
shipping with ext4 and possibly even the default filesystem. Ted
has been using ext4 as his primary filesystem for over 6 months now.
ext4 has also atracted new developers and that of course leads to a
few new bugs as the new developers are less familiar with the constaints
and caveats of the ext3/ext4 body of code.

Jonathan: looking beyond ext4, when is btrfs (pronounced Butter FS or
sometimes just Butter) available (a common question for Chris Mason ;-).
Ted: btrfs is an exciting alternative but doesn't yet compare to the
four decades of experience behind the berkeley style ext3/4 family
of filesystems. It still has some work to be ready for production and
will probably be the follow on filesystem after ext4.

Jonathan: Are there too many filesystems? Ted: Some of the filesystems
are somewhat special purposed, e.g. for flash support or other unique
hardware configuration. However only about 7-8 filesystems make up
about 95% of the total customer base of filesytems in use today.

Andrew was key in shepherding in the a fs - but no particular insight
on what benefits it provides, although the code is very cleanly done and
appears as though it will be very well maintained.

Linux-next: is that working out well? akpm: Yes! It is doing a lot
of the work that he used to need to do for integrationn, testing, and
evaluation of new code. Stephen's work is helping tremendously, although
Andrew feels that the code base is not getting tested by as many
people as it should be.

Where are the biggest problems in Linus' tree coming from? Andrew:
typically they seem to be code that has skipped over linux-next and
gone straight to Linus' tree. That seems to be a bad model and more
people should be planning for including first in linux-next.

Should people be developing against linux-next? Andrew: probably not.
The code base is really not stable enough for that and the various git
trees are not not really well set up for this.

Are there too many developers? Has the rate of change decreased? Andrew:
no, not really. There seems to be a trend of established developers not
always seeing the changes from new developers that make it into the kernel.
In some cases an established developer will stumble across a new
directory in the source tree and find that the code is filled with newbie
mistakes. While this has the potential to be a problem in the long term,
it seems like the openness of the tree is helping to maintain the quality and as subsystems are used and encounter bugs or problems they still get fixed by the community.

Linux-next is causing a lot of email about merge conflicts between subsystems. Is that causing a problem and is it too hard to do develop in the kernel now? Answer from several: no, it seems to make it easier and points out the problems soon.

Question from the audience; Everyone pushes new developers to get code upstream. However, many subsystems seem to meet extreme resistence when pushing code upstream, e.g. uprobes, systemtap. Andrew points out that these subsystems are impacting the very core of the kernel and thus are more heavily scrutinized. Someone suggested that the code being pushed upstream would meet less resistence if the code were cleaner and better designed. Is this something that could be better documented for existing developers? utrace probably has a better chance of getting merged now that several core kernel developers are helping shepherd the code. In many cases, core kernel code being pushed by non-core kernel developers requires a level of responsiveness that those non-core kernel deveopers typically do not respond to. Questions about locking, API changes, overlapping capabilities, implications on other subsystems, etc. need to be answered and code review comments need to be agressively addressed by the developer to have a chance of adoption. Write and Post and never respond will clearly never get code into mainline/core kernel.

Some have suggested that your code should be so good that core developers want to pull your code into the kernel rather than having you push your code into the kernel.

Are there too many tracers in the kernel already? Is anyone actually using any of the tracers? Ftrace alone has probably a dozen tracers built into it. However, most of the documentation for tracing is only in the git logs for the code checkins, which is pretty pathetic.

Question from the audience: it seems that there are lots of functions duplicated in the architecture specific trees. How does one test all architectures for factoring out common code like that? Andrew: there is a linux-arch mailing list which is the contact point for all architecture maintainers to see this type of common factoring. Or, send to Andrew and he will send them out to the architecture maintainers until they stick.

Question from the audience: Things change rapidly, including drivers moving to different directories. Greg KH: the rate of change is still increasing at a linear rate. The usb subsystem changed from 2.6.10 to today, for instance. Is that rate of change going to continue? GregKH: Yes - that was a several year period of time and things change rapidly. Git logs and such help track those changes but things will continue to change.

Question from Christine Hansen: How are the highly experienced (aggregate 100 years of Linux use?) developers mentoring new developers? GregKH: I think about this a lot and we document more, we train people how to accept patches, contribute patches, etc. Christine: There is still a world-wide perception that most of the development is centered around Portland, OR, USA. At some point, all of you n the panel are going to Ascend. To a Mountain. But who will replace you? Is there anything done on the mailing list or within corporations? Andrew: Oleg Nastrov from Russia is a good example of an up and coming individual. And is an example of how someone can rise from nowhere and become a proficient developer. But the community seems to evolve - including people with great capabilities who disappear to other jobs, etc. Keith: Corporations have an ability to do internal mentoring driven in part by the corporations need to build and develop new engineers which helps with some of this within the ecosystem. Keith points out that the financial incentive for some programmers also help them fit into this mentoring environment. (My observation: corporations help make the best of any given individual, but Linux really draws extraordinary developers which often come from unexpected backgrounds. As one of my old mentors and teachers said, "Great Engineers are born, not trained. You can improve a good engineer with training and improve a great engineer with experience, but there is no replacement for a naturally excellent engineer).

A question on interface stability: As an example, iptables as a command needs to remain relatively stable but the kernel interfaces are rarely used by anything other than a small number of applications. Therefore an interlock between the key applications and the kernel is sufficient. There are some cases where this is just a plain old hard problem, such as X11 interfaces for mode setting in kernel - when a user level application does its own mode setting, you wind up with "impossible" conditions which can lead to a kernel crash. Ideally, the APIs would be stable enough for the applications but they should not have to be locked forever preventing new evolutions of a subsytem. Application interfaces tend to stay very stable but are handled on a case by case basis. If you can find a way for new applications to use new interfaces and old applications to continue to survive or to be updated to the new interface, we can over time migrate to a new interface and ultimately retire an API. BTW, this same level of stability is not applied to the in-kernel APIs since all providers and consumers of an API can be updated simultaneously in the kernel source thus obviating the need for long term compatibility interfaces. And, of course, good interface design enabling extensibility is ideal, although it is also impossible to envision all ways that an interface may need to evolve over time.

Ted: I don't know how to get the latest X server onto my distro (Greg: Get a Better Distro!) but some of these challenges are just inherent in the interlock between applications and the kernel.

Question from the audience: Where is new code coming from? Vendors, Hobbyists, etc.? Andrew: Seeing a lot of involvement from vendors to support their new hardware, e.g. Texas Instruments. Greg KH, over 20% of all kernel changes can still not be tracked back to a specific vendor. Ted: In the filesystem space, we still see a lot of people "scratching their own itch" - fixing the one thing that really bothers them. That often comes from non-corporate sponsored users trying to solve problems that annoy them, including university students, home users, etc.

GregKH collects per-release contributor information and feeds that to Jonathan and the LWN site to allow open tracking of the source of contributions.

A good session - very well received by the audience. After this, we are all off to lunch for a while. If my battery lasts, I'll continue tracking sessions.

Final comment from Jim Zemlin (with sustained applause) the LF is awarding an "unsung hero" award. The recipient is Andrew Morton. Andrew happens to be an avid racer and the Linux Foundation has arranged for a track day for Andrew. Only condition is to not get killed at the track.

Andrew tells a story about Mark Merlin and a Ferrari where he forgot to brake on the track and had a near-miss. Oops - we hope he remembers the break on his track day! ;)

Intel hands Moblin project to Linux Foundation; Imad Sousou speaks about the project at the LF Collaboration Summit

2009-04-08T10:14:00.001-07:00

Linux Foundation has taken over the Moblin project from Intel.

Imad Sousou speaks about the Moblin project.

lesswatts.org, enablement for Intel platforms, enablement for Linux graphics, his group works on Xen, KVM, etc., entire development team is very impressive - primarily working upstream. Intel really "gets it" when it comes to open source.

Big companies are terrible stewards of large open source projects. IBM's handling of Eclipse is a good example of migrating stewardship outwards to enable a vendor neutral forum.

Rather than creating yet another new open source foundation for Moblin, Intel chose to transition the project to the Linux Foundation.

Linux Foundation has become very well respected and is well funded, especially given the current economic climate.

Moblin Governance: no disruptions to the project; subsystem maintainers and steering committee. Real key is that upstream is the place where anything "real" happens.

Why did Intel create Moblin? Have to understand a bit about how Intel Software development works. Intel's customers are asking for Linux. The Moblin effort really started with the advent of the thinking around the Atom processor. They needed an operating system to support all of the new devices that Jim mentioned in his previous talk.

Moblin would like to contain everything from the OS to the rest of the environment all included so end developers need only create the end user experience for their particular device. "Moblin is about enabling the best experience" "The user experience. The developer experience."

There are open source projects for nearly anything you can imagine. However, where Moblin (and other projects) tend to find gaps are in the seams between applications & open source projects. And that level of integration is where Moblin spends a good deal of its time (and that is of course also the goal of any major Linux distribution).

Moblin v1: A great start but: it lacked integration, it had technology gaps and the user experience was not as good as was desired.

Moblin v2: A clean break from v1, includes fastboot, new connection management, a lot of UI framework and enhancement work.

The Fast boot experience: fast boot is not a patch or a component. It is really about doing the right things right. Parallelizing bloat is still bloat. Booting in 60 seconds is really bad. Asynchronous actions are tools, not goals.

Boot times are now at 5 seconds (moving towards 2 seconds): kernel 1 second, core systems 1 second, graphics subsystem 1 second, GUI 2 seconds.

Next generation User Interfaces: real world versus computers. Will future UIs use standard controls and toolkits? More likely that the UI of the future is Animation Frameworks. Clutter is their focal area - Intel bought the company. Physics and 3D, Rich animations, Fluid and dynamic user interactions, compelling and innovative modern UIs, allows develepers to develop applications using the same techniques that game developers use.

Another area of focus is Connection Management. It's not only wired and wifi. It includes Bluetooth, WiMAX, 3G, Wifi, Ultrawideband, Telephone data (and voice). Needs to be seamless, enable roaming, and enables sharing.

Working on connman, existing solutions not designed with our goals in mind; changes at the time seemed harder than a clean start, separate UI from functionality. See Marcel Holtman for any questions (he's at the collab summit as well).

The Tools: PowerTOP, LatencyTOP, Project Builder, Moblin Image Creator.

There is a moblin track tomorrow all day. The agenda should be on the LF Events site.

Linux Foundation Collaboration Summit keynote notes

2009-04-08T09:56:00.001-07:00

These may be a little ad hoc but here are some quick notes from the Jim Zemlin's welcoming talk on the state of Linux and the Linux Foundation at the third annual Linux Foundation Collaboration Summit.

Jim Zemlin kicked off the conference with an observation that even in tight times over 400 attendees made it to the 3rd annual Collaboration Summit.

Linux is now being used by nearly ever person in the world nearly every day (cue IBM prodigy video) followed by a slide showing a large number of servers and appliances where Linux is used.

3 trends and 3 opportunities...

Trend One: It's the Economy

IDC: 50_% Yes for servers and clients, 25% evaluating, 25% No on evaluating Linux.

Increasing mergers/consolidation driving IT infrastructure consolidation.

Linux brings lower costs, greater simplicity. A recession causes enterprises to re-think fixed cost assumptions iwth Linux on the desktop.

Linux-related software spending is to increase at 2-3X the rate of Unix and Windows in the overall market.

Linux is the primary beneficiary of the recession, growing 2-3 times faster than any other platform.

Trend 2: Redefining the desktop

Is this again "The Year of the Desktop"?

Old desktop, Thinkpad T20, over $1000, crappy battery.

Also had a Motorola "Flip Phone",

Apple iPhone, about same processor (really?) as Thinkpad T20.

HP Mini 1000, 1.6 MHz Atom, 120 GB Storage, wifi, bluetooth, web cam VOIP, $250. Note: Cheaper than the iPhone. Convergence is starting to happen - are phones the clients of the future? Or sub/mini-notebooks?

Is the PDA or desktop or Kindle the new desktop of the future? Is the TV the future? Mobile internet? Is your car the new desktop. Picture of a complex chair with panoramic display. Or will it be a holographic display interface a la MIT's Human Interface lab activities.

Linux can support any of these models today. Linux is a key and fundamental component of all of these interfaces.

Trend Three: The Cloud

It is real. Linux has the vast lead in the cloud computing space. List of companies: mozy, cassatt, flexiscale, opennebula.org, simory, elastra, mosso, dell, google, 3tera, morph, 10gen, salesforce.com, cohestiveFT, IBM, elastichosts, amazon web services, etc. etc.

Linux has the clear and dominating lead in the cloud computing space.

Opportunity #1: Standards

Linux standardization. Cool picture of penguins with different faces.

Major goal is to keep standards open and fair. Trying to avoid a de facto, lock-in "standard" such as what exists today.

Opportunity #2: Unified Defense

Still hearing that Linux may not be safe - a quadrillion dollars of money runs through the Chicago Merchantile Exchange today, all using Linux.

Open Invention network, defensive publications, software freedom law center, linux defenders, patent commons, peer to patent, open source, OSAPA, Post issue peer to patent, Linux Legal Defense Fund all out there, coordinating, making sure that Linux is a safe platform for end users. Working on both defending Linux within the systems as well as evolving the legal environment

Opportunity #3: It's not just about price:

Windows video looking for a laptop supporting his needs, under $1500. Microsoft is competing for the first time on price.

Now story of Zemlin's equivalent shopping experience. Google G1 @ $179, 42" plasma television $699, HP laptop $250, $99 DVR, and bought an OLPC and gave it to a child in need for $199 - all under the $1500 price paid in the microsoft commercial.

Linux is changing entire models from a technical and business sense.

What's going on at the Linux Foundation.

Events -> Web -> Workgroups -> Training -> Legal Defense -> Standards -> Promotion -> Fellows.

See Wikinomoics - #1 example is Linux. And now time for Linux to up the ante on collaboration.

Television training programs with live telepresense at any time, new social networking tools, fellowship programs to enable people to continue to maintain linux.com (kernel.org?).

Picture of 7*9 - 63 large members. This is where the greater Linux ecosystem comes together. New members are welcomed to get involved.

Linux Video Contest

2009-03-30T18:30:00.000-07:00

You know how life becomes so busy that fun things like blogging fall to the wayside? Well, that's where I've been. I have lots of fodder for some decent blogging but it takes something a little more on the fun side to pull me into the blogosphere again: The Linux Foundation's Video Contest. While it is now to late for new entries, there appears to be a decent number of new videos to vote on. And many of them are pretty cute. I haven't seen many that will make the superbowl commercials but the diversity of thought and rational for using Linux makes good food for thought and there are a few that might help people sort of get what it is all about. There is less focus unfortunately on what Linux can really do for the average home user and more on why people choose to participate in the Linux community or why people use Linux and are annoyed with the reliability or cost of other platforms. But for Linux geeks, there are some summaries of why Linux is a great option and maybe, just maybe, the Linux Foundation will do a repeat of this next year and we'll see some maturing of the videos to focus on why mom & pop or a small business or why your family could or should use Linux.

But in the meantime, go ahead and view (and rate!) the vidoes! It is a fun diversion in the middle of a hectic work week!

Automotive Bail Outs or Building IT infrastructure for the future?

2009-01-07T20:06:00.000-08:00

We live in interesting economic times and I can't claim that I'm a big fan of bailing out businesses that have failed, without finding some way of adapting that business to our current economic and social climates. However, this line of thinking on government investment in building not just highways, but IT infrastructure as a means to improve the basis for much of our economy seems quite exciting. As a few of my previous posts indicate, I'm quite interested in cloud computing as a way to improve the ability of businesses to rapidly adapt to changes and to spend more time focusing on the core of the business as opposed to managing IT. This seems like an area where government investment in support of cloud computing and IT infrastructure updates could substantially benefit government organizations, educational institutions, health care and small businesses. The blog entry has some enticing food for thought and a new outlook on a possible solution for improving our economy, increasing employment, and improve the underlying infrastructure on which so many of our businesses today are built.

Food for thought....

Um, Just Who is Managing Your Public Cloud?

2008-08-15T08:52:00.001-07:00

This article summarizes some of the recent bumps in public clouds. While these bumps are inevitable (and unenviable!) in the early stages of a new technology, they do shine the light on the management of the data center. And, as may be obvious, the people that lost their data in one case most likely have no recourse with the holders of that data. In the case of outages, "well, gee, so sorry" is a pretty weak excuse at the moment for problems in managing the public cloud.

My guess is that this will start a bit of a turn towards more conservative cloud management (that loose and free stuff looks good on paper) and that in turn may start to put a little pressure on prices or start to reduce the license/contractual assurances that current cloud providers make available.

Another thing worth noting here, Google and Amazon, two of the biggest cloud providers, have internal architectures that are designed with high availability in mind. These types of outages would not have affected their core operations, typically. However, most applications that are running in their clouds today were not architected for the same style of high availability.

Anyway, I'll continue to assert that issues like this will help foster the drive towards at least initially, private clouds, with a limited subset of workloads moving into the public clouds based on the type of workload.

It is going to be a bumpy take off into these clouds - fasten your seat belt and hope that the people getting sick along the way aren't on your plane...

BTW here are a couple of other links to recent glitches and failures such as the evaporating cloud or "oops, sorry we deleted your cloud". Some are Web 2.0, but a couple are effectively cloud computing providers which have had public failures - in large part because the data centers and applications were not designed for true high availability or had maintenance issues. And, the last of those links (thanks, Brian!) was just the typical human error problem. Even if you don't create your own cloud, you may well want to really know who is managing your cloud and how - at least until we have some higher end service level agreements available.

VMware joins the Linux Foundation

2008-08-13T19:25:00.001-07:00

So Cloud Computing is the rage today, it is based on virtualization. Many claim that Linux and Open Source was the master key that opened the door to Cloud Computing. So, it seems very fitting that VMware has joined the Linux Foundation. The recent re-/free-pricing of VMware ESX definitely helps make core virtualization a commodity and thus makes it easier to build the more complex software solutions that ultimately will simplify information technology management over the next several years. Linux with VMware ESX, Xen, KVM, etc. now provide a powerful base platform on which to build more complex solutions which will ultimately enrich our lives and reduce the amount of time we spend managing our IT infrastructure.

Welcome to the Linux Foundation, VMware!

You want to participate in an open source development community?

2008-08-13T19:19:00.001-07:00

Then read this: http://ldn.linuxfoundation.org/book/how-participate-linux-community . Kudos to Jonathan Corbet of lwn.net fame. A very good (and relatively short) booklet on how to participate in an open source community. Specifically, this is geared towards Linux, but many of the observations in here will span communities and relate to any development project which is developed where a mailing list is the primary communication medium for developers.

Definitely a good read!

How Secure is your Public Cloud, anyway?

2008-08-13T15:39:00.001-07:00

I've been chatting with people lately a bit about the rate of uptake and adoption of these so-called "public" clouds. While I'm a big fan of the potential here, they still aren't the right thing for all workloads. There are problems with availability, security, latency, etc. which have not all been resolved. As an example, VMware was recently hit by this bug, and black hats identified some holes in Xen security. And these are surely not the last holes. Sometime around the time I was born, IBM started working with virtualization and providing very high end availability, reliability, security and such. VMware and Xen are much younger cousins which have a lot more growing up to do before they provide the security and isolation of physical machines. Of course, the push for Cloud Computing and ubiquitous virtualization will accellerate the improvements in security and isolation in these more modern hypervisors. But I probably wouldn't be putting my corporate intellectual property on a public cloud just yet. Many other workloads may be just fine but think carefully about what goes out into the public domain, er, cloud, and what you protect with those corporate firewalls.

On the other hand, those corporate firewalls give you some protection if you want to use private clouds inside your enterprise today. Those security holes mean that your own employees might get access to more information than you might have intended, but there are other things, like employment contracts, that give you some control over those types of misuses. And, unintentional access resulting from bugs at least puts your data in the hands of people you generally consider reliable.

How can I get a padded jail?

2008-08-06T17:26:00.001-07:00

Jim Zemlin is a featured speaker and panel coordinator today at LinuxworldExpo. His intro to the panel questions started with the assertion that over the years (have there really been 18 LinuxWorld Expos so far? wow) that Linux has become nearly ubiquitous, including a lot of pictures of mobile devices, servers, desktops, laptops, services, collaborations tools, etc, which are all using Linux. He talked also about initiatives, including Green data centers, Cloud Computing, etc. which are more widely enabled as a result of Linux being so prevalent and accessible within the industry. In many ways, Linux is enabling many of these emerging technologies because it provides a common basis for innovation which is easily accessible and eliminates the need to build every new initiative or product from scratch.

Jim also provided a reinforcement that the "competitor" from which we in the Linux community need to learn from today is no longer Microsoft (well, they might have a trick or two that we can still learn) but the real competitor today is Apple. Jim took a poll to see who has some sort of Apple device today and at first glance, it appeared to be the entire room -- At a Linux conference! -- had an Apple product. A little digging showed that Apple products weren't quite ubiquitous but the point was by then made. Jim also pointed out how Microsoft and Apple are finding a way to sell products that have vendor lock in. The products are not open, not easily available, controlled by a single entity and basically are a jail for consumers. Of course, he then pointed out that the Apple Jail looked a lot like a 4 star hotel room with video on demand, a great view, clean and neat, and was a jail that most of us find to be rather luxurious. The next slide, though showed the Microsoft Jail - emphasizing that the roughness of conditions were exacerbated by the fact that you were often trapped in that jail with no amentities, some very large rough looking malware types, and a raft of viruses to make your stay as unpleasant as possible. And, the wrap up was the equivalent Linux "Jail" is more like a visit to Burning Man - free and open, yeah, there may not be a lot of frills, the power might go out, but you are free to come and go as you will, you can improve your surroundings as you choose, and ultimately you can really enjoy yourself. Perhaps Burning Man is not the best analogy here, but it makes the point quite nicely.

Jim's panelists included James Bottomley of kernel community fame, Christie from the Motorola alliance providing Linux enabled cell phones, and David who helped create the (no longer available in stores) Walmart PC.

Picking the right target for the Linux Desktop

2008-08-06T14:33:00.001-07:00

I think it has long been recognized that having a Linux desktop look "as good as" a Windows desktop has been a pretty low bar from an easy-to-use point of view. Mark Shuttleworth brought it up again at the Linux Symposium during his keynote speech (something he has clearly been thinking about for a while), and I just saw Bob Sutor bring it up again at the Next Generation Data Center keynote speech that he gave.

I think the Linux desktop has gotten a lot closer to the simplicity that the Mac offers or that Windows offers, but in either case, it still has a long way to go. There are still so many areas that I've been fighting with on an Ubuntu laptop (T61p at the moment since the display on my tried and true T41p decided to blink out for good last week). Because I have worked with Linux for a long time, I'm relatively confident that with time and enough good google searches I will resolve the problems. But boy do I rue spending the time on realizing that NetworkManager is trying to take over my wireless and doing everything it can to make sure I can never connect to a wireless access point. Or, I can use the nv driver without compiz, or the nvidia driver without suspend/hibernate. Oh, if I dig through various forums, it looks like there are possible fixes/configuration changes that might move me forward, but if those answers are out there, why does an apt-get install not just fix all those problems?

Then there are the annoyances - I put in a USB key on this box and for some reason there is a hard hang of the desktop sometimes for a minute or two and then Nautilus opens, finally. I put the same USB key in the Mac and Finder just opens. My iMac 24 isn't very mobile but when I powered it up at home, I had to choose/enter an SSID, fill in a password and select an authentication mechanism, and through three wireless routers, several reconfigs and such, it just *does the right thing*. The use cases are a little bit different but Linux still doesn't seem to do the right thing.

And these are just the basics - what about all of the more complex, cool tasks. Dragging video, editing it and copy/pasting subsets of audio or video. Managing my music library or managing TV recordings seems to be always "possible" on Linux but never easy (my MythTV stopped last time I lost power and/or had an automated upgrade).

But I think the message remains: Pick the right target for comparison, and that right target is hopefully clearly not Vista, XP or any past Windows product, but is instead the much more user friendly environment of the Mac...

Cloud Computing paper presented at the Linuxsymposium is now available

2008-08-05T15:44:00.001-07:00

Well, I've dropped way behind being "as it happens" with news and info, but my Cloud Computing paper is now available from the linuxsymposium.org site - my paper is about page 197 of Volume 1 I will also make the slides available to anyone that asks (stripped down a little but still a bit chunky because I went overboard on pictures of clouds a bit, oops.)

I'm currently at the Next Generation Data Center - interestingly enough there was a lot of alignment between the Cisco keynote speech and what we are working on in IBM. I'm also sitting in on the Virtualization 2.0 track where there is a lot of discussion about the pain points and progress in moving from virtualization 1.0 to virtuailzation 2.0 (btw, I do not see a crisp definition of the differences from the presenters, most just a view that virtualization is evolving - and rather slowly at that. The most appropriate quote was that the adoption cycles are much slower than the talk-about cycles. But there is clearly progress in adoption of virtualization and some of the new problems I've referred to before are also becoming more visible across the industry, such as consolidation exposing more problems in high availability and such.

All in all, the rate and pass of change related to virtualization, grid, cloud computing (oh, matrix computing came up - I have to look that one up) is constant, although pretty slow in the eyes of the technologists.

Linux Foundation Collaboration Summit, Austin 2008, Day #1

2008-04-09T15:10:00.000-07:00

Have you ever had those weeks where your calendar is continuously double booked, or triple booked? Well, this week has been double booked solidly and quadruply booked sometimes, with the challenge of having some of those meetings being in different cities. This time the cities for my meetings included Beaverton (close to my nice home office in Portland, Oregon), Orlando, and Austin. And, all of them are activities that I normally have a strong interest in. One was IBM's Technical Leadership Event in Orlando, which pulls in some of IBM's top talent and provides best of breed internal training. That was a must-attend event, and I was able to catch the last bit of the next event which was also a must-attend event.

The Linux Foundation's Collaboration Summit, began on Tuesday in Austin. While I wasn't there, at the beginning, many of my counterparts were able to both represent and take some notes so I can get back up to speed. I'll share some of those here since there is some good stuff in here.

First, the program started with the "State of the Kernel" panel. One of the topics focused on kernel quality, where the kernel community seems to see all the problems - there is a lot less visibility to the good things about any given kernel release. However, even without balanced input, the consensus seems to be that the kernel quality has improved over the past year. My own experience seems to reflect that as well in that most kernel updates lately seem to demonstrate that the tricky things like power management, suspend, hibernate, laptop drivers, etc. Normally, I'd check some of the automated kernel test results to help validate my thinking, but it looks like the conversion to autotest has left the answer a little less clear then they used to be but this view comes closer to providing a snapshot of kernel stability. But at a broad subjective level, that consensus appears to reflect the broader consensus that I typically see.

Device drivers were again a topic, although in the past year, a phenomonal step forward in terms of the transitioning from perception of driver problems by the user community and perception of phenomenal driver coverage by the development community (both perceptions were right, of course!) with the creation of the Linux Driver Project and their list of drivers needing love the problem has transitioned from a near pointless debate to a focused activity where users and developers come together to solve the problem. The downside is that there are still a lot of drivers needed but the good news is that people are steadily working their way through the list.

There was a question regarding the Linux Test Project and its interaction with the Linux Kernel Community - the key problem is that LTP sometimes is out of date, covers surface tests, or generates false failures. I know the LTP team is working on that, and the kernel team is interested on working through that. There was a proposal at the kernel summit to include LTP in the kernel tree to help fix and maintain it more aggressively. However, Linus was not too keen on that, so that has been blocked. I don't have a current update on where that sits but would love to know the latest thinking.

And, last for that session, RealTime Linux was cited as a great example of the community coming together to find a consensus solution amongst competing alternative activities and potential solutions. RT is now used in production.

That's probably enough for one entry... Look for more as I find time.

Is Cloud Computing nothing but Utility Computing?

2008-03-14T16:20:00.000-07:00

So there's this, ahem, interesting blog entry over at RedMonk by James Governer. His assertion on 15 Ways to Tell Its Not Cloud Computing really seems to refer to utility computing rather than cloud computing. And utility computing has been around for a while and matches pretty much all of his assertions. Except for some reason, providers of utility computing, and, worse, consumers of utility computing are still hard to find. Oh, they exist, Amazon EC2 provides effectively a utility computing model, other major vendors have various offerings in this space. But that is not what Cloud Computing is.

More importantly, Cloud Computing provides utility computing, either for a business, or for an internet community. Utility computing is an output, but somewhere underneath that utility there is a lot of coal mining, a lot of power plants, a lot of sub stations that provide that utility computing. And, despite the long time goals of the Grid community, the performance and security challenges (and a few others) haven't allowed for effective sharing of computing power between companies. And, to be honest, even the paradigm shift to Cloud Computing is not going to bring us to that Nirvana over night. Even with some of the greatest computing shifts, change occurs over several years, and with several steps that allow the industry to keep up and to adapt to those paradigm shifts.

As an example, I have been a part of IBM's Linux strategy since about 1999. I've also been involved heavily with the Linux and several open source communities. One realization there was that Linux was going to transform the desktop first, then data center, and then the world. Well, we had the order wrong. And we had the order wrong because the place where change happened on this scale was actually in the Data Center. And that is exactly where the change for Cloud Computing will come from first. And, if you look at the 15 tests to see which ones matter to Enterprises, you'll see that most of them just don't matter. And that is not because Enterprise's don't want utility computing today - it is because they still want a high level of control and management over their data.

I would assert that of the Fortune 500 or Global 1000, there are very few companies putting their trust in an Amazon EC2 offering, nor would they do so for a very long time. And, for small mom & pop shops, they may start with EC2 as a cheap entry point, but there is a point where they need more control, more flexibility, more capability than they can get from today's utility computing providers. And that is the point where, having tasted the services that a Cloud can offer, they will want to build their own, with their own requirements built into it. (Yes, there are more of the Fortune 500 that outsource compute resources, but I would assert that for the most part, those are not clouds for various reasons too lengthy to get into here).

So, if you've read James' blog, here are a few brief comments: I'd agree with James on OGSA, on provisioning/deprovisioning (although Cloud Computing is *sooo* much more than that, see one of my earlier blog entries on that). Well, I thought there'd be more that I agreed with. But I don't disagree with all the rest - but I don't think any of those have any basis in determining whether or not something is or is not a cloud.

Maybe this is part of the core: Amazon (and google, and others) have provided a very visible, very interesting model which demonstrates that underneath the utility computing, they have built a cloud. They maintain that cloud, they provide the services, the manpower (Human IT resources), the intelligence of reducing the cost of managing that cloud, the data centers, the power management, the service and hardware and software maintenance of that cloud. The end users benefit from (and pay for) the utility provided by that cloud.

But what about those companies that want all of the best practices, the simplification of management, the ability to create their own virtual appliances matching their workload, their ability to manage Sorbanes-Oaxley requirements and do it in house? Those are the companies that today are interested in buying into the Cloud Computing vision, not just as a consumer of some of its services, but as companies who push the envelope on innovation as a way to fuel their own business.

Virtualization and consolidation: Less Availability and Less Predictable Performance?

2008-03-10T13:32:00.001-07:00

Oh No - say it isn't so! Will virtualization and consolidation put more workloads on the same hardware? When that hardware fails, won't it take out more of your workloads? Remind me, how did consolidation help me? Oh, and with more workloads running on single machines, what happens if all my workloads are busy and growing busier all the time? Won't my performance and throughput go down?

Well, in a nutshell, yes. Yes, that is, if all you do is squeeze workloads onto a smaller number of machines with no thought to the aspects of performance, availability, and throughput. Even with sites such as Amazon's EC2 or AWS, there is currently no published availability plan as of this post on the O'Reilly radar. I think Bill Hammon's post here also gives a concise view of the problem statement. In Bill's post, he addresses two of the key issues - availability and performance that tend to accompany consolidation. I think there are several others as well, but these make a good start. The problem is that, as with any new technology, consolidation and virtualization introduce some new challenges. In this case, base provisioning, management of an image catalog, and simple management of virtual images make consolidation look easy. Basically, you transition your current tools to a virtualized environment, and presto! you are well on your way towards cost savings, ease of management, additional easy-to-access compute cycles for additional test and development activities. But, most people still manage each of those virtual machines as they would a physical machine, and we run more virtual machines on each physical machine - potentially say, 10 virtual machines per physical machine. That means we have to manage N + 10 * N machines, oh, and after consolidation, most people don't throw away the previous machines, the plan to re-use them as their workload grows. Add to that the fact that most workloads being consolidated today are those infrastructure or web services workloads where the availability story is based on running multiple servers of the same type and failover handled by something as simple as DNS or a first-to-respond policy. Consolidating all of your scale out capacity on a few machines means that your scale out policy is no longer going to work as you expect.

On the performance side of things, you have something of the opposite problem. Where previously you would have capacity for anywhere from twice to ten times (or more) the typical average workload, and sometimes as much as twice the capacity needed for peak performance, you now have the case where the aggregate of your ten virtual machines may run at 50% utilization on average, with peak capacity still twice your average. But now when two or three or four virtual machines are using something well in access of their average, the performance of the entire set of virtual machines may be impacted. Today, most workloads which are being consolidated don't find that limit to be too onerous, but as more critical workloads move into the consolidated environment, the risk of oversubscribed physical machines will grow.

So how do you combat these trends? Consolidation, despite these challenges, it still a huge potential money saver. One of the comments in the O'Reilly radar link above suggested one solution: pair your internal infrastructure with a hosting service like Amazon's, and provide availability that way. Clever, although your degraded mode of operation may not be as predictable in terms of responsiveness and performance, at least you have a degraded mode of operation. Another solution is to build an HA solution into your virtual machines and manage your HA yourself. Of course, if all of Amazon's EC2 is done (or any other provider) that doesn't help you much.

A better solution might be to look at the problem in terms of Cloud Computing (yeah, you saw this coming, I bet! :-). Within a cloud, you should conceptually be able to say "instantiate my virtual machine, and I'd like some of these properties." "These properties" might include some level of HA, some level of performance/throughput guarantee (an SLA), some level of backup, maybe even some concept of energy management/efficiency. So, if you have your own in-house cloud and a catalog of virtual machine images that you typically deploy, you would be able to specify your desired level of HA - how highly available and at what cost; your desired performance - what cap on machine resources you want, if any, or some level of guarantee of end-to-end throughput or latency; whether you could contribute your data center's "green" goals by using energy efficient hardware resources, etc.

And, part of the point of a good "Cloud" implementation would be that it would handle these details for you. The best practices of HA would be patterns that could be deployed based in part on a high level specification such as "4 nines" or "5 nines" of availability, or performance could be managed by a workload manager actively migrating tasks within your pool of resources as needed. And no, that isn't technology that is 5 years out... Cloud Computing is a potentially disruptive technology in development right now...

Open Grid Forum: Grids and Clouds

2008-03-05T17:35:00.000-08:00

I recently heard about Irving Wladowsky-Berger's keynote speech at the Open Grid Forum and was pleased to see that the slides from his keynote were made available there as well. While I'd love to have the transcript from that talk available - Irving is a very engaging and dynamic speaker, the slides alone are also quite interesting. In particular, Slides 19 and 20 provide a nice little visual on something we are referring to as Ensembles, which I'll talk about more in the future. But the graphics give an interesting preview of the thinking regarding how we an simplify the data center. Yeah, they need the transcript or some discussion to enlighten the reader but I think it visually plants a useful concept on grouping of like resources as a means of simplifying the management of those resources. There are some other resources on grids and cloud computing which I think are worth reading, including some comments from Steve Crumb, the Executive Director of the Open Grid Form, and from Ian Foster who is a visionary and leader in the Grid computing space.

The short summary is: Clouds are more than Grids. But there are similarities and in some ways, Clouds build upon some of the thinking and concepts inherent in Grid Computing. Of course, Clouds build on a number of concepts, include autonomic computing, on-demand computing, virtualization, self-healing, and many of the other trends over the past several years. I think the biggest difference over many of the past views is that previously these technologies were focused on improving aspects of the computing environment - where cloud computing really focused on bringing those strategies together to provide value to the end user and to reduce the cost and effort of managing computing resources.

Of course, not all of the work is done, cloud computing is not something you can buy off the shelf today, and some of the work requires a paradigm shift across the industry. On the other hand, most of the work of cloud computing will be done by standing on the shoulders of other giant technology leaps in the industry, so in part, much of the work will be to integrate and reshape those technologies into a new and more accessible form. The journey promises to be challenging but the reward appears to be great!

Feet in the Data Center but Head in the Clouds

2008-03-05T17:09:00.000-08:00

Okay, you get it, I'm interested in Cloud Computing... And I have a few things to share on that in general. But the last few entries are really focused on using cloud computing as a solution to fixing what is growing into a mounting crisis, or at least a danged annoying trend of spending more time managing our computers than using them. But there's also another reason that clouds are very interesting, and I think this summary on MSNBC of the work that Google and IBM are doing to start training the next generation of thinkers and problem solvers.

In part, the core of the article is really about thinking about how to use our compute resources to more effectively solve problems. I'm reading over and over about the time it takes for someone to start a new project inside a business. Usually the first step is to find hardware - and enough hardware - to do something interesting. Then there is the location, the configuration, the network access, the account management, oh, what software did you need installed, by the way is there enough power there? Is your new project mission critical? Did you think about backups? Do you need something more reliable? And soon, the mere thought of creating a new project gets plowed under by the gnarly logistics of just getting prepared for a project.

The cool thinking in this google driven model is that it transforms compute power the same way that the Internet transformed our connectedness. Keep in mind that networks are this complex mass of ethernet and switches and hubs and WANs and LANs and firewalls and wireless and cable and goodness knows what else. In some way, the complexity of configuration is high - not as high as servers perhaps, but still not trivial. And, errors like having the power cord kicked out from a switch in a lab may only take down one of my IRC servers for a while, while the rest of the internet remains connected (yes, that happened today). But we don't yet think of compute power as a utility, nor do we think of it as a plentiful utility, which it truly is. If we added up all of the compute power currently in operation on the planet, well, the overall ability is staggering. Perhaps as staggering as the sheer volumes of data running around the Internet today, every day. And with the internet, it sends email, pictures, provides services, all without us really noticing the underlying utilities. We all have access to that staggering amount of network bandwidth, but we don't generally have access to that level of compute power, even though the vast majority of compute power in the world is seriously underutilized.

Google's thinking - and IBM's, Yahoo (er, MicroSoft), Amazon, etc., is that it is time to make these large data farms more accessible. And, to do so requires several large shifts along the way. One shift is to provide some of the fundamentals for managing the servers at a way that reduces the impact on the environment (or corporate pocket book) as well as the human cost for administering these systems. Another, and the focus of the joint education program that Google and IBM are embarking on, is to start educating people on ways that they can more effectively make use of the compute power - not just as a single machine, but as a utility which can be harnessed for increasingly larger challenges.

My day job is focused on building up the infrastructure to simplify the server environment management and ease the access to that server capacity. But in the long run, the shifts in education, the shifts in programming model, will arrive hopefully just as we have mastered the ability to deploy large cloud computing environments. It is definitely something to look forward to!

Is Cloud Computing just Provisioning?

2008-03-03T11:29:00.000-08:00

There is more and more buzz on Cloud Computing (and, it just happens, that is why I've been too busy to blog lately ;-). However, one of the common assumptions that seems to be floating around is that Cloud Computing is nothing but a provisioning exercise. As an example (btw, an example of some pretty cool technology) is this reference to Cohesive Flexible Technologies which does some cool stuff with provisioning. Or, 3tera has a great demo for provisioning.

But provisioning is only a part of the story, and, in some ways, only the beginning of the story. There are several other key elements that make cloud computing ultimately more valuable in a business sense. For instance, virtualization is almost a requirement for great cloud computing. Several solutions are already based heavily on virtualization, but most provisioning solutions to date have been focused on deployments on physical hardware, with a few looking at deploying on virtual hardware. Why is virtualization important? Hmm, short answer is that it allows consolidation, migration, isolation, security, and several serviceability requirements than increase the overall value of the cloud. I may go into more depth on that in a future article because the benefits may not all be as visible on the surface.

Another part of the story is image managment, which today is less well evolved than it ultimately needs to be. I've mentioned rPath & rBuilder previously, Amazon provides views of images, but we are basically in that early stage of image management where the number of solutions and repositories tends to boggle the mind. This portion of cloud computing will need some additional standardization, improved tooling, better life cycle management, etc. and will be key to deploying solutions within clouds.

Management of the life cycle of applications or images (or virtual appliances, as some call them) within the cloud is also a challenging area in which there is very little product available today. For instance, people tend to deploy singletons or redundant sets of servers or images. But HA hasn't done a large merge yet with cloud computing. For instance, it should be possible to deploy a virtual appliance which "rarely fails" or "never fails" from a customer point of view. HA solutions can be crafted that way today, but usually they are hand tuned, hand configured. In a true cloud computing environment, that should be merely a parameter to the deployment.

Other aspects include handling things preventive maintenance - does your service go down whenever someone needs to install a new service pack or update the hardware? In a cloud environment, there should be other resources your application could use - so why not just migrate your application to those resources without disruption? Oh, but how do you handle migrating network connections, how do you handle access to the license server? Is someone collecting usage information for charge back? How is that usage information integrated across the cloud for charge back? How do you measure and manage your response times for multi-tiered applications?

I think all of these are aspects that will eventually be included in the expectations of things that a Cloud just handles. And, some of the services that a Cloud will soon offer will cover those bases and potentially many more. Clouds are much more than just provisioning.

Simplifying IT

2008-01-02T14:38:00.000-08:00

It seems like it has been ages since my last blog entry. Little things like a new project, a vacation, some holidays all seemed to pull me into new and interesting things, most not involving a computer. And while I thought about blogging from the beach in Maui, there always seemed to be a little umbrella drink between me and my Blackberry (although now I really understand why many refer to them as "crackberries"). But, at last the new year rolls around and I'm working myself back into my normal routine. Slowly. With help from Starbucks. Luckily, a wise few chose to extend their holidays through the end of the week so I could return to my labors of love slowly, gently, and with a little time to blog.

In our last entry, I was sitting alternately in meetings about Green Computing, about large, google-sized farms of computers, about complexities of management, about strategies for simplifying management of large farms, about costs of machine configuration, about the human cost for managing complex environments, about the costs for securing, installing, updating, configuring, reconfiguring, administering, relocating, cooling, powering, virtualizing, migrating, reconfiguring again, replacing, modernizing, consolidating, and, oh, then my head exploded and I wrote a blog entry.

Now, all these things don't seem to be isolated events, but most of these problems seem to be showing up at nearly every major data center. A few data centers managers are very proud that they have installed computers 5 times faster than their old ones, and thus reduced costs, or they have installed the latest high powered cooling systems, or installed the largest data center powered directly from the Hoover Dam and thus, they project, they have managed to avert another data center crisis for a year or two.

Of course, most improvements have come at the cost of the environment, or perhaps the data centers fix (which doubled the power consumption for the company) doesn't show that they increased their annual power bill by more than the cost of all of the upgrades. The most common solution seems to be to add more of your favorite dual processor Intel or AMD based system, some with a whopping two or four processors, each supported by several fans, various power supplies, several co-processors, a variety of PCI cards, oh, and gigabytes and gigabytes of memory per machine (memory gobbles up what, now, up to about 20% of your power in some configurations?) Yet the workloads continue to double or triple in size every year. That is nearly an order of magnitude in compute power every three to four years. On average, that increase in workload seems to exceed Moore's Law, meaning that simple upgrades of processors and pure technology replacement is never going to keep pace with the growing rate of compute/storage/network consumption.

Vendors are working hard to address parts of the problem, as I alluded to in the last article. Yes, processors are getting faster and memories are getting more dense. Cooling doors for cabinets are taking out some heat. Virtualization is starting to improve efficiency, but at the cost of increased complexity for systems administrators. All of these improvements combined do not seem to be keeping up with the explosive rate of growth in terms of consumption of compute cycles.

I think we are due (overdue, in fact) for a paradigm shift.

I believe we have many technologies at hand which have been poised for a while to provide some relief from this fast growing cycle. There is even a term for this paradigm shift that several large companies are starting to use. While the meaning of the term is still fairly open and ambiguous, I believe it pulls together some basic principles hold some promise for alleviating many of today's pain points in computing. Specifically, this paradigm shift is being called "Cloud Computing" and provides a framework which, over time, can help alleviate not only the rapid growth of workloads, but also be a focal point for energy management, for abstraction of workloads from physical hardware, and ultimately, for the overall simplification of IT.

Green Computing: Environmental Concern or IT Crisis?

2007-11-21T17:39:00.000-08:00

I've been buried in a number of projects lately, but as things slow down before the US Thanksgiving holiday, I've had a chance to peruse a number of bookmarks I've been tagging related to topics that I'm working on. And, I'm seeing some interesting trends linking many of these together which seems to be at the nexus for a number of new technologies and announcements. Looking at these patterns may help provide some focus understanding which of the emerging technologies has the greatest chances of having a significant impact on the IT market.

Of the trends, one of the most common right now is the constantly increasing price of oil, the global tension surrounding oil, its impacts on the global economy, and the ramifications on the overall cost of energy. Energy prices are obviously one factor leading to the nearly universal concern in data centers that energy prices are becoming a significant cost factor. Obviously, the rate of growth of data centers, the trend towards installing more cheap Intel based servers, the explosive growth of new workloads and better linkages between people and information are also rapidly consuming available power.

Related to this problem is of course the quantity of heat generated by all of these servers and the requirement that servers operate within a relatively narrow temperature range. This means that data center cooling is another source of power consumption. And, it means that air conditioning and cooling has to be fit into the same floor space as the servers. Since it raises the power consumption bill as well, it becomes a more frequently measured source of IT expenses. And, the dilemma of how to add more cooling and more compute power in the same, typically existing data centers, is causing many big IT organizations to build new IT data centers.
Of course, for those with some sort of conscience about the environmental factors at work here, this overall IT crisis is also becoming a Green concern. The amount of energy consumed, the amount of heat generated, the amount of often "prime" space being used for data centers is globally becoming enough to stagger the mind. And, the rate at which the new and innovative workloads is increasing is going to continue this trend in a superlinear fashion - I can't predict that it is an exponential growth because of containing parameters such as cost, floor space, the economy, power availability, human labor and related IT staffing, etc. But I can safely say from my personal experience that all signs point to something substantially greater than a linear increase in workload growth and data center sizes.

Another trend that seems to be prevalent in data centers is that, for many reasons, there has been a many decade trend towards heterogeneity, driven by many factors. Obviously, pricing/cost is a significant factor. Innovations happening at different rates in different companies also provides a value at some times beyond cost (those high value items that drive the margin behind any company with staying power in the industry). Personal preferences by the IT staff, relationships between company representatives and sales teams, interoperability of complementary products, etc. - the list goes on and on - all drive decisions which ultimately lead to the deployment of highly heterogeneous environments. In many ways, that heterogeneity of the data center drives a Darwinistic survival of the best/fittest environments, which can easily be viewed as a great thing.

However, this heterogeneity in the data center is also driving increasing complexity, increasing costs for interoperability, and, ultimately increasing IT management costs. In some ways, this last point might be one of the most expensive for most data centers. The cost of each person hired to manage data centers often exceeds the cost of the acquisition of those machines, and over time becomes one of the dominant expenses. While Systems Administrators might rebel at my line of thinking here - basically that paying people to administer systems is becoming a disproportionate cost of doing business - the reality is that this spending on administrating the infrastructure is taking money away from a company's ability to invest in its own unique business value.

In summary, I believe that we are approaching an IT crisis here, which happens to align with the increasingly popular concerns with the environment. Those points of crisis are centered around data center issues of power consumption and availability, the increasing need for data center cooling, the unconstrained growth of data centers, and the uncontrolled costs of IT management. While there are many solutions attempting to address components of these problems, such as the previously reported on linuxpowertop.org, or new (new again?) chilled water cooling doors for racks, improved air flow or power efficiency in a compute cabinate, smarter data center layouts, improved systems management tooling, etc., I don't think any of these changes will be enough to resolve the superlinear problem of the growing IT crisis. However, I do think there are a few things on the horizon that have a chance of making a more significant difference, and hopefully over the holidays I'll have time to put up another entry that goes into some of the changes I believe are necessary to address this IT crisis.

Ottawa Linux Symposium 2008: Call For Papers

2007-11-16T18:44:00.000-08:00

The Call for Papers for OLS, or, better to be known as "The Linux Symposium" since this will be the last year it is held in Ottawa, came out yesterday. This year the committee is going to go all out for the 10th anniversary of the conference and expects to have multiple keynote speakers in addition to the normal BOFs, presentations and tuturials. And, the facilitators will provide space for mini-summits before the actual conference. And, the program committee is working to expand the topics to include additional areas of innovation around Linux. These might include creative uses of virtualization, as opposed to strictly the implementation of virtualization, mechanisms for improved power management, overall improvements to desktop or handheld usability, or new technologies that are emerging in the Linux ecosystem.

The Call For Papers is online: http://www.linuxsymposium.org/2008/cfp.php.

The 2008 Linux Symposium should be the best symposium yet! Make sure to submit your papers early and register early as prices rise closer to the actual conference.

Linux Device Support

2007-10-23T17:26:00.000-07:00

The Linux Desktop survey from the Linux Foundation is now in full swing. I also Chair the Vendor Advisory Council for the Linux Foundation and recently did a poll of the vendors to identify the top 10 issues inhibiting Linux adoption from the vendors point of view. A recent meeting of the Linux Foundation's Desktop group with the three primary vendors that now pre-install Linux also identified a number of key issues to address. I also have met with the User Advisory Council of the Linux Foundation a number of times to hear their issues. And, there are always surveys online and summaries of installation tests, primarily on desktops.

In all of these sources of end user pain points, I always hear about how device drivers and device support ranks in the top 5 (sometimes top 1 or top 2) issues from each of these groups. I also have some first hand evidence of drivers as a problem, especially when dealing with the ultra-latest in hardware support. At IBM, we set up a "tiger team" to help identify new hardware, work with independent hardware vendors to provide them encouragement and training on how to develop open source drivers. When I was part of OSDL's Data Center Linux effort, I helped develop training information for developers, managers, project managers on how to work drivers into Linux. And, with help from the Linux community, OSDL's legal team, and the Linux Foundation after that, we set up programs to allow vendors to enter into NDA's with Linux driver writers. The goal all along has been to enable vendors to get drivers written, either on their own, or with help from the greater community of Linux driver writers and to have those drivers exist as part of the Linux kernel, available to all users when they install their Linux distribution. And, the Linux community itself has made available over 200 driver writers and 10 project managers to help create device drivers.

Yes, there are holdouts like nVidia who have not embraced the open source driver path. There are challenges in the wireless space related to the FCC operational ranges for transcievers. But by and large, the inhibitors to developing and making device drivers available for hardware have been removed. So, I am somewhat puzzled by the fact that what was clearly the #1 inhibitor to Linux adoption 5 years ago is still on the list as an inhibitor. And, I'm struggling now to find out why. As part of the Linux Foundation, I've pledged to dig into details and attempt to provide concrete lists of devices which need drivers, and honestly, I need help with this. I no longer have a list of devices that are not supported by Linux, and, if I had such a list, I would work all possible channels to get drivers for those devices written and into mainline. And, the list has to be more concrete than some of these comments from the desktop survey:

hardware problems (USB, bluetooth devices - webcams, headsets, ...)
Difficulty connecting with some devices.
Drivers Support for some devices (Printers, WLAN, ..).

There are some better comments in there, such as:

The need to manually compile drivers for external devices like a usb tv tuner (for example AverMedia provides the driver but there is no user friendly tool to install it and u need console commands knowledge and internet access in order to complete the install)
Quality drivers. Non-official drivers usually support only a subset of the full capabilities of hardware devices, and even official ones (like the Intel graphics driver, for instance) have a lot of annoying bugs.

But even those are a bit vague. There is also some confusion between "devices" such as printers and scanners (many of which have pass through kernel drivers with user mode application drivers) and those that have a full kernel driver to enable their capability. Of course, that is something that the desktop group can work through over time.

But my primary point here is that getting detail about *what* devices do not have support has been getting much, much harder. People need to provide information about Vendor, Product, Model number, PCI ID, USB ID, etc. to help the community identify what drivers are broken. And, when a driver is missing capability, as is alluded to in the "quality drivers" comment, the missing features need to be identified. Until we have that list of non-functioning or limited-function device, there isn't much that people can do. And, sometimes it is key to continuously let people know that features or devices are not working from release to release. The community often moves so quickly that something broken one day could be fixed the very next, and without a periodic reassessment on the functionality of the device is sometimes necessary.

Oh, and before someone points out the obvious ones, yes, nVidia now has the nouveau driver, ATI has started working with the community to support their graphics devices, Intel is working towards full support of their hardware, most SCSI and SAS devices are currently supported, and there are some rough edges still with MultiPath support that are still in progress. But what else is broken? Surely these few devices can not be the substantial reason that device support is constantly on the top of the list? What other devices do not have driver support? Expose them! That is the best way to get those devices supported!!!

Ballmer out fear-mongering again

2007-10-10T09:35:00.000-07:00

Ballmer is out telling the Microsoft faithful that Red Hat customers owe Microsoft compensation. Of course, he's not clear on what people owe Microsoft for, other than some vague allegation of "intellectual property." The details, of course, more like a Hitchcock film - cue the ominous music and make sure the camera is focused where you can't actually see anything. That always brings out the greatest fear in humans - that fear of something truly frightening that you can't actually see but are only led to belief that it must exist. But, just like with Hollywood, there seems to be nothing behind these allegations. Slide the camera down a few feet and you'll see the fake blood, the poor props, and the people with no wounds, much like the scene that Microsoft seems to have set up. Some props, some large scary numbers, a lot of repeating how scary it is, and of course, a lot of press to propogate a lot of nothing.

This big number came out originally around May and sounded oh so frightful. Of course what very few people looked at was the density of patents around a single piece of technology. For instance, 15 patents supposedly were implemented amongst several email programs. Of course, it could have been 15 email program each of which *might* have happened to implement something that bore some resemblence to one of the alleged patent. And, nothing looks at the scope of those alleged patents - are they for turning a high priority message's subject Bright Red? Gee, that's easy to work around or fix. Is it for some mailbox flag being raised when new mail arrives? And gee, if MS patented that, was there prior are back during the beginnings of the internet, back before MS was a player in the email field? And, let's assume there *is* some patent that might, just might be implemented by some mailer. And that's a big IF from what I've seen thus far, but assuming there were one, do you know what would happen in the Open Source community? Step 1: Remove the feature. Step 2: Implement it better.

So, what are MS's options with this dubious set of FUD (Fear, Uncertainty and Doubt) that they are creating? Maybe they should sue some rich people who are using Linux, like the good Mr. Ballmer suggest. Oh, wait, look at that - most of those rich Linux customers are also running Microsoft products. OH, what kind of PR does *that* make, when you sue your own customers? I bet that's a great way to make them run from your company. Others speculated that MS could use the strategy of just ignoring those patents and IP. Of course, MS could never do something so simple, could they? No, what they are doing and going to continue to do is play the ominous Hitchcock movie music, wave a lot of props, jump around in the press exclaiming how scary this should be for everyone, and then extort "insurance" money in the form of cross licensing to put to rest their competitors' fears (yes, those same fears that they helped create, that have no substantiated basis).

I kind of wish someone could just tell MS to put up (the goods) or shut up. Playing on people's fears without any basis is really just immature and going to continue to cause their reputation to be affiliated with the dark side of the force.

Security: Objects vs. Names

2007-10-03T15:27:00.000-07:00

Emily writes about the annual kerfluffle known as LSM vs. SELinux and I've been reading the coverage on slashdot and kerneltrap with some amusement lately. Last year this was again coming to a head around the time of the Linux Kernel Summit and one of the topics was to have the AppArmour/LSM folks and the SELinux folks get together to talk about how to resolve their differences. And, in the end, they were unable to resolve them, although the reason is that LSM and SELinux take two different approaches to security - both valid, but roughly incompatible.

SELinux was started by the NSA and provides an appropriate level of security for people that are really paranoid about the security of specific "objects" - files, directories, pipes, shared memory, etc. The SELinux people strongly believe that the goal is to protect an object by all means necessary, and all permissions are ultimately associated with that object. This allows for both Discretionary Access Controls (where I can give you access to my directory or file with chmod(1) or some form of Access Control List (ACL), and for Mandatory Access Controls (someone gave me a file which requires certain permissions, and I can not open that file up to someone who does not have those permissions - think of it more like Classified documents). With SELinux, all objects have their own security traits and they are completely unrelated to the name or location of that file.

This obviously sounds like good stuff for security; however, the downside is that the implementation of object based security has been a bit more invasive on the kernel side, very difficult to use and configure from the system administrator side, and started life in early deployments as the feature that was always shut off to get it out of the way. In other words, it often crossed the line between usefulness and security. This is where that age old comment that the only secure system is the one that is unplugged in a locked room buried under ten feet of concrete" comes in. Usability typically drops as security goes up.

LSM provides another mechanism related implementing security. It starts with the premise that every object has a name. That seems simple enough, right? In fact, most of us don't talk about the file that holds all of the kernel messages, but instead we refer to /var/log/messages (or your system's equivalent) as the place where logs are stored. In fact, pretty much every interesting object has to have a name - and probably a name in the filesystem (there are some exceptions to this but that isn't really important here). And, LSM works from the premise that systems administrators know the names of the files that they want to protect or manage. Therefore, why not provide a security mechanism which takes as its primary "handle" for accessing an object a filesystem path name? And, from that small shift in fundamental focus, SELinux and LSM tend to diverge while both providing very strong, or sufficiently strong system security mechanism.

A more interesting difference between the two shows up when one wants to audit all actions on a file. Let's say I'm monitoring /var/log/auth.log watching to see who becomes root on my system. In SELinux, I might monitor all reads and writes to /var/log/auth.log. But what if someone moved /var/log/auth.log to /var/log/auth.hide? SELinux would continue to monitor my moved object for changes, even though it isn't the place where the current notification of reads and writes was happening. And, just supposed that some black hat at the end removed the new /var/log/auth.log and put back the /var/log/auth.hide as the new log? It might appear as though the entries of the black hat hacker just disappeared! Of course, we've monitored all reads/writes to /var/log/auth.log and the audit logs may not show the problem. This is a case where LSM track actions on the path /var/log/auth.log no matter what object it pointed to. Of course, this case is a bit contrived, but shows the point. In reality, we'd probably monitor all reads, writes, moves, truncates, creates, etc. of the object in question and in both SELinux and while using LSM we'd be able to track the black hat.

One final point regarding the pluggable debate - SELinux allows a single, fixed mechanism with some policy extensions. If it doesn't have the right capabilities or permission controls, well, you just can't monitor something. LSM provides a pluggable/semi-stackable (I'll stay out of the politics of this one at the moment) interface, which means that you can create more complex rules for denying access to files. One part of the debate is whether or not LSM should have that flexibility - what if you stack a function which doesn't just permit or deny access, but instead adds extra functionality? Who monitors that functionality to ensure that it does not in some way corrupt the kernel or make some locking layering violation, or simply changes some existing priority scheme by allowing access to a file inappropriately? The problem with flexibility is that it is like the rope you can use to hang yourself or the rope to build a more comfortable hammock. There is more concern about people using hooks to do the former (like in Spiderman - "with great power comes great responsibility").

But yet again (and probably indefinitely) *both* security policies are in to stay, while people play with both until some new mechanism emerges that is better than either.

Linux Kernel Summit 2008

2007-09-21T12:48:00.000-07:00

Well, it isn't official yet, but it looks good that the Linux Kernel Summit will be in Portland in 2008. The program committee for the kernel summit pretty unanimously sees it as a good idea to adjoin the conference with the Linux Plumber's Conference (covered in an early entry). This should help out the Plumber's Conference as well as encourage collaboration across the various projects.

Growing Linux Desktop market share

2007-09-21T08:43:00.000-07:00

I have been reading this article over at Information Week over and over the past few days because it bugs me. It bugs me for several reasons, the most important reason being that part of it is right. Now, I think a few of the "7 reasons" are bogus and irrelevent but a few are close to correct. And, the premise that this is not the Year of the Linux Desktop, and that next year is probably not the Year of the Linux Desktop is valid. And it won't be the Year of the Linux Desktop until a few things change. The good news is that some of those things that need to change have started or are slowly inching forward. But there are still a few issues which were touched upon in that article that have not yet been addressed at all.

First, let me do a quick (very quick) commentary on the key points from that article, followed by a few thoughts on what is missing. Alex Wolfe's 7 points are (slightly simplified):

1. Prohibitive application porting costs
2. Fanboy alienation
3. You can't make money on the operating system
4. Resistence from average users
5. Linux is "simple"; Windows "just works"
6. There are way too many Linux Distros
7. No Powerful Evangelist

I think of these, the three that are most important are Applications (and Porting), the "simple" vs. "just works" distinction, and, a bit, the comment about too many Linux Distros. I'll come back to those three in just a moment.

With respect to Fanboy alienation - most end users don't deal with Fanboys, or, if they do, those Fanboys are just as prevalent for any OS, and software, any hardware. And face it, most of these fanboys are die-hard geeks who learn their social skills in SecondLife. I don't think that is helping or hindering Linux in any material way. As far as making money on the OS - OS's haven't been a profit center for anyone *except* Microsoft for over 10 years now, maybe closer to 15. In the days of the Unix wars, the investment in the OS was high but the derived value was always in the enablement of hardware or applications. That surely hasn't changed. As relates to resistence from average users, there may be a bit of that because people resist paradigm changes that don't add value to their lives. And, in many cases, Linux provides "equivalents" for Windows applications, not clearly superior applications. And, of course, there are many applications missing or simply hard to find/identify. Especially when free software doesn't wind up in boxes that people can pick up at Costco or Walmart saying "here's how to do your taxes". And, as far as evangelism, Linux grass roots evangelism seems to be as powerful as any force out there. And, of course, the Linux Foundation is working to provide a focal point for some of that evangelism as well.

However, the three points that I believe are valid from above have slightly different core issues. Yes, the absence of common, key applications on Linux is a challenge. I still struggle with my Linux x86-64 platform because Sun Java plugins for Firefox are not available, Apple's QuickTime isn't available, Macromedia isn't available, etc. Yes, there are workarounds, but they are not bundled in a single "yum" package with dependencies that allows me to "just make it work". But there are other applications that don't provide native Linux ports today - TurboTax, PhotoShop (sure, gimp is more powerful, perhaps, but as a photo editing novice, I was able to do more with PhotoShop in a few hours than I've been able to do with the gimp in several days), and other desktop software. On the other hand, nearly every major application that runs on Windows servers has a strong equivalent or a direct port to Linux, such as databases, web servers, PHP, perl, python, backup/restore software, etc.

I think the real problem here is that interactive applications may be a bit more difficult to write for Linux, in part because things like sound, graphics, panels for print, print preview, document layout, etc. have so many options for the developer to choose from. The Portland Project is helping to standardize many of these with the goal of simplifying the writing/porting of desktop applications. This is one excellent activity that came out of OSDL/The Linux Foundation pulling together the various Linux Desktop Architects and a number of desktop ISVs. Of course, reaping the full benefits from this activity are still a year or two out. I could drone on more about the challenges here and some of the other activities (like the Linux Standard Base) but I'll leave those for another time. The summary though is that there are some challenges in writing/porting desktop applications to Linux, although usually it isn't porting cost that stops companies from having Linux applications, but the ongoing maintainance and support costs based on diverging platforms, changes in the underlying environment, and the required QA that needs to go into the release.

I see a parallel between Linux and IBM internally that parallels Alex's "simple" vs. "just works" argument, although I'd phrase it a bit differently. Linux (and IBM) have *great* technology that can solve nearly any problem, address nearly any workload. However, translating that technology into desktop value, business value requires an expert in those technologies. IBM has started delivering "Express" solutions which provide less functionality that is more accessible, and in some sense, that is what the various distros try to do for a given target end user. But, that leads to the last point that I wanted to address, and what may actually be the most important overall.

I believe there *are* too many Linux distros. Well, maybe not "too many" exactly, but too many that are too different in all of the minutia that matter to application writers, end users, systems administrators, etc. And, most of those differences do not add any value to any of the distros - pulling in a slightly newer version of one library here that is incompatible with some other application or library that has several new features, or changing the location of repositories or the directory structure to match one distros view of "the best" location. Many of those types of changes wind up creating fragmented support forums on a per distro basis (how do I get my TV tuner card to work? Well, what distro and what version of that distro are you running?). And, this proliferation of distros makes it especially hard for the ISV's writing software to know how to adapt to a given distro. Sure, the LSB helps to address some of those, but the LSB is like a color by number book, and to be honest, most cool applications are closer to Van Gogh (or maybe Picasso) when it comes to creativity, and that creativity often requires capabilities beyond the LSB.

My radical answer to this problem, which might also solve some of the desktop adoption issues: Start with a single, common distro as the core for those hundreds of other distros. Something like Fedora or Ubuntu, which are already pretty common. What if Fedora were the basis for OpenSuSE and Ubuntu, for instance, and then those distros added their value *above* the basic capabilities that most ISV's ported to? As we pointed out above, most of the desktop distros are given away for free anyway - there is no value add returned for all that investment in the desktop, typically. The real value is still being added above some basic/core level. Why not consolidate on a single base building block core and give up the rat race of adding negative value at great development cost to many independent distros, and focus on the value add above that line?

[John Cherry had some excellent comments (below) and also posted an article on his blog at the Linux Foundation which is worth reading!]

New Linux Conference: Linux Plumbers

2007-09-13T21:15:00.000-07:00

The word for a while has been that Portland is a hot bed of Linux and Open Source developers. It looks like they have decided to put that concentrated open source expertise to a new use: a conference dedicated to developers, with the goal of collaborating around some of the remaining hard problems. Oh, and, no, the conference is not exclusive to Portlanders or web-toed Pacific Northerwest denizens. In fact, the hope is that the conference will draw developers into the northwest to draw on the rich expertise available and also provide access to a wider cross section of developers that is present at some of the other conferences.

One might ask, though, is one more conference one conference too many?

My answer: I don't think so. There *are* a lot of conferences and events out there today: LinuxWorld Expo (for C* level execs, possibly dying as Linux reaches mainstream corporate adoption), Ottawa Linux Symposium (hitting its 10th anniversary, going strong as a place to present current work as a user, a developer, or a systems administrator), the Linux Kernel Summit (invitation only, Linux kernel centric), Linux.conf.au (a great conference, but hey, it is way over there down under and all!), the new Linux.conf.eu (great for the europeans, if still a little small), Linux Kongress (supplanted by the Kernel Summit and Linux.conf.eu this year, back again in 2008).

Whew, that's a lot of Linux boondoggle travelling for the well funded Linux (or Open Source) geek. However, note that the only one of those in the US is for people that probably never even install Linux, but ultimately decide if it has business benefits for them. The only other one in North America is targetted rather broadly at presentations of current activities in the kernel and nearby user level, often by the experts in those fields. While OLS provides BoF's (Birds of a Feather) session for people to chat about common interests, there is no drive or mission to actually converge on solutions at OLS (although that does happen on occasion).

What the Plumber's conference seems to provide is a forum for what I refer to as the "mini-summits" - places where developers can get together to work out key issues. This seems to be a new need and a new phenomena in Open Source, where email, IRC, and mailing lists are king. However, not all issues can actually be resolved in forums where often the loudest or most prolific people can sometimes set the opinion for a group, or where some issues cycle without ending because the collaboration mechanism doesn't seem to drive consensus on tougher issues.

The Plumbers conference should allow developers to talk face to face and work through some of those harder issues, much like often happens inside of companies doing proprietary development today. If you've worked in a proprietary environment, especially one where the development team is local, you've certainly seen the value of pulling the whole team together to hash out issues. I think setting up a scenario where the community can do the equivalent of the "team meeting" - especially for those cross project issues can only help with the overall integration of Linux and increase the value to end users.

Great blog on the Legal mechanisms in place to protect Linux

2007-09-13T18:47:00.000-07:00

Andy Updegrove posted a good blog with the reasons behind the upcoming Linux Foundation sponsered legal summits. I highly recommend reading it. It demonstrates one of many methods in progress designed to better develop the legal ecosystem around Linux and to clarify the general legal status of Linux and Open Source. I happen to work at IBM and as many know, IBM seems to hire more lawyers than just about anyone else. And, it tends to hire some very skilled people into those roles. I think many of these lawyers have been in the vanguard of understanding the implications of various open source licenses and their potential impacts on IBM business. Yet, I occasionally encounter lawyers who haven't had the pleasure of working with open source and are at least initially confounded by the differences in the various licenses which surround open source code.

And, having seen how these new laywers come to terms with those issues, often with input from seasoned lawyers, experienced technical and managerial folks, I occasionally wonder how a laywer without that support network would come up to speed on this complex new world that open source provides. From what I've seen thus far, new doctors of jurisprudence rarely get in depth training in school about a ground breaking license like the GPL. And, with the variety of opinions and interpretions often held by all of the arm chair lawyers that the GPL has created, as well as the political and business pressures around open source, I'm not surprised when laywers take the "safe" route of simply recommending "no" when open source comes up.

However, for those businesses that take that simple "no" response and choose to stay out of the open source world, I believe that in many cases they will artificially limit the success of their business operations. Picture, for instance, a world where IBM in 1999 said "no" to Linux, because we were unable to understand the impacts of this upstart GPL license. Sure, someone else would have sooner or later endorsed Linux, in part because the allure to business is so strong, but also because an open minded lawyer actually can come to terms with the GPL. And, to be honest, the legal environment really isn't any more complex than the normal legal wranglings that any large company goes through - it is just that the rules are a bit different, and that in the end, all of the sources released are visible to anyone that cares to look. The extra scrutiny scares laywers (and sometimes developers who are worried about code quality ;-) but in the end, that extra scrutiny also helps ensure that everyone stays within the terms of the respective licenses.

The Linux Foundation's Legal Summits will hopefully blow away some of that fear, uncertainty and doubt, that smokey haze of confusion surrounding open source for those that are less experienced, and allow the legal teams of companies using open source to make informed decisions. Beyond that, it will help companies set up appropriate isolation to prevent cross contamination of source bases when using incompatible licenses (most companies do this today even for non-Open Source licenses) and to understand their rights and obligations when using open source as consumers.

All in all, I see these Legal Summits as a great value add to the Linux ecosystem!

Linux Kernel Summit: Customer Panel slides

2007-09-13T13:06:00.001-07:00

The slides for the three presentations made at the Linux kernel summit are available here:

Sean Kamath, Dreamworks
Head Bubba, from Credit Suisse
Markus Rex, The Linux Foundation.

They provide a bit more detail than what I included in my blog. Sean also added comments in the powerpoint/openoffice version which I'll try to put on the site as well.

Amanda at the Linux Foundation also put up a short blog entry about the event.

Microsoft disses Google on code quality

2007-09-11T21:34:00.000-07:00

Okay, I just ran into this article on Google which gave me a couple of laughs. The first was this quote: "...companies should question the actual number of users Google has 'within the enterprise'". Well, I *think* Microsoft was suggesting that there might not be many Google users within an enterprise, and if so, someone must be seriously deluded! I don't know numbers or anything (although I'm sure they are out there) but I bet there are close to as many "Google users" in an enterprise as there are Microsoft users. Of course, many are probably using primarily the search capabilities, but this question could backfire on them if they aren't careful. The other funny point was to beware of 'Google's history of releasing "incomplete products, calling them beta software"'. Well, Duh, wake up and smell the coffee! There is a new trend of people who, for non-business critical apps, are more than willing to use some classes of software before they are "done" - and, in most of those cases, people have the benefit of requesting features and bug fixes which actually get applied in a timely fashion, rather than in a product to be released in three years.

Now, I'm not saying evolutionary development of software with rapid deployment, a la an Agile or Lean development method is right for all software or all business uses, but Hey - the waterfall model is dead, and getting user input during the design and development phase of a product, with rapid responses to that input leads to better products, sooner! Being offended because the world is moving on to something more dynamic is not going to win the heart and soul of businesses that are driven to quickly provide the best services to their employees and customers.

The real issue here is that an old product development method is being challenged by a new development model. And, while I expect that both models will coexist for a long time, anyone burying their head in the sand and dissing the new model because the old is "better" needs to wakeup. Now.

Linux Kernel Summit: Future Directions

2007-09-10T12:22:00.000-07:00

The last formal session for the kernel summit was to look at the logistics for the next summit. There was quite a bit of discussion about the incestuous nature of the program committee, the elite attitude that the summit appeared to create, and some question of the pedigree of some of the vendors and committee people on the kernel summit discussion list. It makes for amusing reading at times, but some of the concerns needed to be raised and discussed in person. To help improve the neutrality of the discussion, Ted Ts'o introduced the topic but then asked Dave Miller to moderate. Dave happens to be among the most highly respected people of the core, wasnot on the committee the past couple of years but has served on the committee and understands the challenges that such a committee has. As such, he was able to moderate with an eye towards representing the community point of view. I captured a number of the questions/comments raised during that session. Not all questions raised had answers, but many did, and the net summary is that the view is that the program committee is actually doing a good job at pleasing most people most of the time (the highest achievable goal, in many people's eyes ;).

Here are some of those comments, captured mostly via stream of consciousness discussion.

Process Issues vs Technical Discussion
- How was this year? Too much on process issues?

- Holding BOFs in the evening was a bit much
- Is three days a good idea: NO
- BOFs in mid day would be good
- Process was good to have here.
- This has been far more productive than it has been in several years
- Linus still likes the customer/user feedback
- Linus might like to have more (two) different customer sessions?
- Linus really likes the feedback on the process stuff here/face to face
- Hugh strongly agreed on the customer panel.
- Dave Miller pointed out that the customer panels are often
so negative; hearing both good and bad as good
- Benh found it to be amazing

Number of attendees: too large? Too small?
- No larger
- 12 person committee seems large
- Ensures less bias in voting
- Provides shepherds for some of the topic areas
- Most would be invited anyone - Dave Jones doesn't think they
auto-invites are bad
- the invite-only nature causes issues
- Limiting the invitations seems critical; otherwise too many people,
lots of vocal fans
- Elite-ism versus openness issue
- Co-locating with another conference adds value in conjunction with
invite-only
- Important discussions seem to still happen in the short breaks
- Proposal for some smaller discussion groups/parallel panels
- Counterproposal - no small groups (hch)
- Mini-summits before the conference
- HCH strongly believes this should not be panels and lots of
people but should foster cross-polination.
- linux.conf.eu - several attendees who weren't burned out by KS
- KS + OLS usually leads to burnout by end of OLS
- Some went to a pre-minisummit + KS
- Alan Cox - much smaller and you have just "the usual suspects"
and then nothing useful gets done.
- Quite a few new faces - first timers, which is viewed as good
- 5 slot lottery of Maintainers is viewed as a good thing
- Should there be more people here representing lower levels of
user level, e.g. glibc? (they were invited and didn't show)
- After the first round, consider an appeal process for round #2,
per Steve Hemminger?
- Andrew Morton - it is easy for people to fall through the cracks
- Matt Mackall - 2,000 people on the committer list - did we invite
enough of them?
- Andrew - should we consider moving to every other year? Or Dave
Howells - maybe every 18 months.
- Holding this every other year has a corporate downside - it might
fall out of budget.
- Russell King - how well is embedded represented? about 10 people
Would also like to also see some other well-known arm/embedded
people present (Cambridge is the home of ARM, hence would have
expected more people here).
- Should take into account whether someone is local a bit, including
perhaps local funding.
- David Howells - going to Japan might bring in local embedded developers
then, for instance.
- At LCA there will also be a kernel mini-summit.
- Generally people seem to be pretty happy with the program committee.

Organizing committee:
- Picked to have adequate knowledge/coverage of kernel subsystems,
corporate/distro funded work, etc. to pick invitees and topics

- How is the PC chosen? Ted has been chairing and has picked the
program committee. Ted is trying to pick people who have good
coverage of all of the different parts of the kernel, as well as
providing some corporate insight. People who contribute actual
work to the program committee are more likely to get invited
back. People who are interested in being on the program committee
need to let Ted know.

Location next year?

(Here Ted showed some statistics from a survey he did about the time of this years OLS, numbers are from those statistics):

Will people attend KS in US? 12% object, 88% no object
Will people attend KS in Canada? 3% object, 97% no object
Should we coalese KS with another conference? 10% very important, 46% somewhat important, 44% does not matter
Location is much more important for sponsors who care about travel budget.
Approximately 87/188 responses 46% response rate
How many are likely to attend OLS? 18% will, 27% likely, 21% attend if co-located with KS 31% not likely to attend, 4% will not attend

Possible future locations, ranked by interest:
Vancouver
Australia
San Fran
Portland
Seattle
San Diego
Boston
Ottawa

No European Location listed/ request to ping/pong between Europe/US)
We talked about India/Asia/South America as options; very few attendees from there, thus travel long distance for everyone. Dave Miller strongly objects to ruling out China, for instance.

Possible locations considered (including co-location with another conference):

Ottawa w/ OLS (3rd week in July, 2008)
Portland w/ Plumbers Conf (September 2008) - Kristin
Someplace else? LCA Early 2009

Strong votes for going with FOSS in Bangalore
Also recommended China + some conference.

During all of this discussion, Dave Miller was a very strong advocate of taking kernel summit and/or developers to Asia (China, Japan, Korea, etc.) as well as India, South America, and other developing areas to help build a strong global contingency. The general leadership consensus seems to be that there are huge pools of talent to be tapped in those areas, as well as significant Linux usage, but that development and use of Linux are not well represented by the current activities on linux-kernel. Usually building face to face relationships helps expedite and encourage this sort of contribution and should be highly encouraged through both the kernel summit, related conferences, individual developers participating, and through the Linux Foundation's outreach program.

And that took us to the warm beer (did you know you could find Budweiser in Cambridge, UK? I thought England was more or less famous for either its beers or its beer drinking, but come on! Budweiser?!) and then the photo session.

I also recommend checking out the coverage on lwn.net by Jonathan Corbet - he brings insights to his reporting that are well worth the subscription fees for early access, and, if the trivial cost is too great, all of his articles become public after a week. (shameless plug for someone who's editorial and reporting skills I greatly respect!).

Linux Kernel Summit: Andrew Morton on Kernel Development

2007-09-10T12:15:00.000-07:00

Andrew used a session at the kernel summit to provide general advice to linux kernel developers. Most of the device was targetted at the current process and focused on the representative leadership at the summit, although this advice should be heeded by anyone submitting code to the Linux kernel. Linus frequently jumped in with his own advice and support for Andrew. Here are a few of the excerpts that I captured.

Quilt/Git tree maintainers - please re-sync your tree as soon as you merge with Linus' tree.

At last year's kernel summit, maintainers agreed to send out a one-page summary of what they expected to submit in the next release. Kudos for Roland for actually doing this. The rest of you, please step up to the challenge.

Large patches like code cleanups, white space, lindent, code motion, etc. are proposed to go in right after -rc1. When those are applied and cleaned up - that will probably lead to -rc2. The x86 transmorgification would not be a good example for this, though.

Akpm sends out patches to subsystems maintainers which are only 30% or so applied or rejected by maintainers. Roland requested more clarity in the directions with those emails. Andrew clarified that cc: means review, To: line to the maintainer means to merge it or approve/reject it.

Andrew prefers to merge his 1000 patches *after* submitting the subsystem maintainer trees. Everyone thus usually gets two weeks, and Andrew gets 24 hours to merge 1000 patches. Linus is tired of subsystem mainatiners asking for merges in the last week (last day?) of the merge window. Subsystem maintainers who claim that there are conflicts in the merges are usually wrong and the merge should not take them so long. Linus wants subsystem trees (especially git trees) in the first DAY of the merge window.

Linus also really hates having people say "please pull my git tree" during -rc2 or -rc3. He tends to flame extensively for such travesties, primarily because otherwise he might just pull the tree is silent frustration. In other cases, Linus will silently no longer pull trees if they are outside the merge window. Subsystem maintainers should be enforcing this just like Linus does. Linus was *quite* blunt in his assertion of this policy and gently encouraged (with a large hammer and four letter words) that people help enforce this policy.

ACPI maintainers received kudos for hitting the merge window perfectly in terms of the previously described policy. The networking maintainer (Dave Miller) has been, um, a little more iffy lately, Linus said, somewhat tongue in cheek. All rules, of course, should be interpreted within the guidelines of common sense. Linus suggested that white space cleanups and such should then go into the end of the merge window.

Andrew pointed out that the -mm trees are now once every two weeks or so, as opposed to two a week. He plans to step that rate up again. Mel finds that the -mm tree is usually stable within 2-3 days and can be run on his machines by then. Andrew had promised to try to get daily snapshots out originally and gave up on it because the patchsets were actually seldom stable until he was ready to publish. Ben H. was asking for access to the patches that were going into -mm more frequently, even if the full tree was not functional. The -mm-commits list contains all of these patches as they are applied to the -mm tree.

Andrew believes that there should be some way for the kernel community to deliver user level applications to kernel users. There are about 15 .c files in the Documentation directory. It is not clear yet what the boundaries for inclusion in such a code directory. util-linux package used to be the place for all this per Christoph Helwig. Christoph will help move all the .c files from Documentation into util-linux. Balbir pointed out that there were some example code in Documentation; Christoph pointed out that that was the wrong place for it to be. He felt that util-linux would be the right place even to store basic examples. Maintainers is now Karel Zak (from RedHat) who forked util-linux which was previously unmaintained. The fork is called util-linux-ng.

Andrew was wondering if we were doing a good job of getting things back into the stable tree. He thinks that he is sending about 90% of the right stuff to the stable tree. Greg KH recommends adding stable@kernel.org to the cc: list of any bug fixes that are sent to Linux/Andrew so that they will see when these bugs are committed. There are 80-90 subsystem maintainers - Andrew encourages all of them to be considering which patches should be going to the stable tree.

Semantic checkers such as Sparse and Coverity are good and finding problems; however, some people are doing code motion rather than intelligent evaluation of the code. His example was tests for NULL at the end of a function are often getting moved up in the function, which may be correct, which shuts off the sparse/coverity warning, but in reality, the ideal patch may be to remove that test if that case really can not happen. In other words, use intelligence when fixing problems found by automated tools!

Linux Kernel Summit: Containers Update

2007-09-10T10:24:00.000-07:00

Containers is a significant new development activity in the Linux kernel today. And, it has drawn together a wide variety of independent implementations (out of kernel, today) working towards a common goal. Representing the broader community on the Kernel Summit update was Paul Menage of Google and Eric Biederman, known previously for his work on kexec and kdump, amongst other things. The discussion started with some highlights about containers, including the rather well-agreed to concept of resource namespaces. Namespaces provide a means of grouping various resources, such as process (task) ID's, IPC's, network connections, etc. into discrete "containers" which are can be isolated from other namespaces. Ideally, applications running in a container will only see the tasks and activities related to that application in their view of the world. Also, these individual namespaces can have their resources controlled independently and one containers resource usage should not impact the resource usage of another container. Today, the UTS (utsname) and System V IPC namespace isolation mechanisms are in mainline. The pid namespace is being testined in the -mm tree and is one of the more complex namespaces to isolate. The networking namespace work has complete prototypes, but there are still discussions in progress on how this might go forward in the Linux networking space. Some have suggested that the current Level 2 and level 3 isolation mechanisms are overkill and that netlink offers a mechanism for doing much of this today. The jury is still out on the details of how this will make it to mainline, but hopefully these will be addressed over the next couple of months. Other resources which still need work include time (virtual time, primarily, I believe) and devices, such as pty's. There is also an open question as to whether there are enough CLONE_ bits for the clone() system call given all these new inheriticance properties for use during the creation of a new process or task.

Paul talked a bit about the renaming of his current work to be Control Groups - primarily because his use of the term "containers" overlapped with many of the terms used by other containers groups and both teams were getting confused. Paul talked about CFS and the ability to apply CPU weights to arbitrary groups of processes, about cpusets and some of the rework he has had to do (Andrew Morton volunteered Paul to be the cpusets maintainer since it appears Paul Jackson has taken a sabbatical from Linux Kernel development), about the memory controller work that Balbir Singh is doing, and talked a bit about the problems with the task freezer for freezing and unfreezing tasks. He also mentioned an NSProxy as a way to tie namespaces to control group, and talked about how aggregated limits and controls for swap, disk IO, dirty pages, and network restrictions could be done.

Paul also addressed the common questions about: Well, Why not just use existing interfaces such as.... setrlimit: which can only restrict tasks to simple numerical limits, with no generic support for aggregate limits, and which are only settable on the current process; or uid/gid/pgrp/session concepts: which have historical semantics which can not be co-opted, can only (generally) be set on the current process, and can't be set to arbitrary values. In other words, they don't have much value in allowing system administrators to group and manage processes and their resource consumption.

Additional benefits include the fact that control groups can be nested; they form a strict hierarchy. And, their semantics when nested will depend on the specific resource controller used. Paul pointed out that while the framework has no real measureable performance impact, various resource controllers may trade off throughput for quality of services.

When people asked about the size of the overall namespace code, the answer was that mostly existing lines of code are modified, with a minor addition of code on top of the modifications.

When people asked how close containers are to being able to replicate the functinality of Solaris Zones, the most significant missing component today is the networking code. Once that is worked out, the differences in capability between Zones and Containers is relatively small. And, chances are that Containers will provide some flexibility above what Zones provides in the foreseeable future.

Linux Kernel Summit: Memory Management

2007-09-06T14:26:00.000-07:00

Next Peter Zijlstra and Mel Gorman led a memory management session. There was some discussion of variable page sizes, with Linus expressing a strong preference for continuing the hugetlbfs strategy, including allowing creation of separate hugetlbfs segments of distinct, native hardware page sizes. In other words, an administrator would create potentially a large page region of 1 GB pages and another of 16 GB pages on a platform that supported those two page sizes. Then the applications would be responsible for explicitly accessing the page regions of the appropriate size. There were several people arguing strenously for a more automated usage of various large pages sizes without requiring application awareness, such as is done in Solaris. Linus would have none of that, arguing that the complexity and maintainability of the MM subsystem would be severely compromised and there would be new and unpleasant application side effects when large pages were not available. The impacts on MM locking in general would probably ripple widely throughout the kernel. The largest concession that Linus made was that extracting common library functions from the mainline path and the hugetlbfs code to simplify maintainance would not bother him at all - a very pragmatic assessment.

The other issue that Mel confronted was that memory management workloads tend to vary widely and to be generally complex. Plus, the various MM contributors seem to be very inconsistent when requiring testing to validate a new patch or change in that area. Specifically, Mel tried to get a clear answer to what type of workload or testing would be acceptable to all in validating performance related patches against the MM subsystem. There was never a clear answer to the question, mostly with the answer "it depends" or "it depends on the patch". But no workload was validated as a "good" workload and microbenchmarks were pretty clearly not the right answer for most of the MM patches. There was an expectation that people working in this space would submit performance comparison numbers for some valid workload showing the value of the patch, but Mel responded that most workloads have been rebuffed with the response that they are not reasonable or valid workloads.

I don't think that conflict was ever resolved, and was left as an open "do the right thing" sort of response. It will be interesting to see how the next few conflicts like that are resolved on LKML or linux-mm.

Linux Kernel Summit: Scalability (and Embedded)

2007-09-06T14:18:00.000-07:00

Nick Piggin and Matt Mackall led the discussion on scalability - both to larger AND smaller machines. Nick didn't have a lot of detail to discuss on scaling up. In general, Linux today addresses many of the key scalability issues, although there were a couple mentioned during an earlier session related to filesystem recovery speeds that still need to be addressed. Specifically, ext3 filesystems (or ext4) over 16 TB or so are taking much longer to recover, and as filesystems grow, the time to repair after a critical failure can be unreasonably long. However, Nick mostly focused on memory & processor scalability, and there was some mention of some bottleneck in block IO scalability that I didn't catch, although it sounded like it was possibly already being addressed.

Matt Mackall mentioned a few things about the embedded world and the kernel footprint which seems to be growing at a reasonable pace. Apparently the pressure to keep the cache footprint for large systems small has some synergy with the embedded needs for a small memory footprint.

Generally, though, this session didn't have a lot of material and finished early.

Linux Kernel Summit: Real Time and Scheduler

2007-09-06T14:04:00.000-07:00

So, I missed part of the real time and scheduler discussion since I was out in the hallway discussing SystemTap's challenges with Christoph Helwig. SystemTap specifically is depending on utrace which is not yet in kernel and which has some challenges to be addressed. I got some insight into what the problems are and will see if there are things that the SystemTap team can do to help address those. Anyway... back to the topic at hand, as I re-joined it in progress...

Anyway, Thomas Gleixner, Ingo Molnar and Zach Brown led the session (Thomas and Ingo being primarily focused on real time kernel support, and Ingo and Zach working on syslets). Thomas and Ingo were requested additional testing & usage of the RT kernel by more people, especially those that had additional workloads which might benefit from real time kernel support. Real time has made a lot of progress lately and is running some very intensive real time workloads, including being the basis for the next generation US Navy Destroyer class ship and having been deployed on laser wielding robots (I think Thomas mentioned that they just shipped the 250th laser weilding robot with Real Time Linux) -- OH - and those are the GOOD laser weilding robots - not the bad ones! Mostly these are used in manufacturing for welding and such, not for sci fi horror movies. :)

The other topic, somewhat related, is the syslet and general asynch system call operation, such as is required for good operation of asynchronous IO, e.g. AIO. Suparna pointed out that the corner cases for AIO were tough originally, and there is probably more work to do in that space, although the original AIO work addressed many of the original complexities.

Kernel Summit: Customer Panel

2007-09-06T06:54:00.000-07:00

Thursday's first kernel summit session was a customer panel, organized by James Bottomley and yours truly. The presenters were Sean Kamath from Dreamworks, Head Bubba from Credit Suisse, and Markus Rex, the new CTO for The Linux Foundation. Sean covered the basics of their configuration and workloads, then pointed out some of the key problems that they see in their IT environment around Linux. These problems centered around memory management and a recent shift in semantics regarding RO/RW mounts of local and NFS filesystems. Some of the memory management problems were reportedly fixed (or at least possibly fixed) since the latest enterprise distribution that most customer are using. However, a number of the questions centered around information that was very difficult to for a corporate IT department to determine definitively and, with some reticence, the group present recognized that as a possible problem. Specifically, understanding what capabilities are available for handling out of memory conditions, how to limit the resident set of a process (or group of processes), and how to determine how much memory is being used by a process (or group of processes) were considered difficult for end users to determine. Further, some of these behaviors are different between versions of mainline and the various distro releases.

Head Bubba began with an intro to Credit Suisse's business and IT environment. He then followed with an explanation of some of the key problems that he sees in his environment. The first observation was that nearly every change to the CPU scheduler has an impact on the behavior of many applications. While those impacts can be easily contained when an application writer knows they are coming, but often those changes are made either silently, or only with notification to LKML, which isn't really sufficient for end users (complementing lots of discussion about the signal to noise ratio on LKML is leading to many fewer readers of LKML lately).

Head Bubba's next mentioned the value of Linux Real Time which is currently in development, and some measurements based on his workload and the benefits seen by using RT. One large area of pain is general access to Linux diagnostic tools. SystemTap was mentioned in particular as was utrace and PAPI. Bubba then reviewed some problems with jitter in TCP/IP. The last area where Bubba focused was iWarp and OpenFabrics and the performance, and, more importantly, the latency of packets received in these high bandwidth interconnect technologies. There are some interesting charts in the slides that describe these problems.

The last speaker was Markus Rex, covering the consolidated list of priorities from the Linux Foundations User and Vendor Advisory Councils. The list started with a high level set of areas that are of interest to these groups, including such obvious things as virtualization, power management, scalability, IPv6 readiness, etc. Markus then drilled into some of the specifics in each area, although the attendees would have preferred to see more specifics in some areas, such as scalability. Within virtualization, for instance, it was most important to be able to run any OS as a guest on any major distribution. Right now the paravirt_ops is generally moving in that direction, but the major distros are still not quite there. The hope is that a server or workstation could support running a plurality of distributions as guests simultaneously. Today we aren't quite there with out of the box distributions, although that will likely be present in the next versions of the enterprise distros and most concurrent distros in that time frame. And, with all that flexibliity, end users are going to want to have a single, standardized management interface to all that capability, e.g. CIM interfaces and a unified management application.

Other requested features include power management capabilities for servers, and increased support for device drivers, in conjunction with the community plans for enabling open source device drivers.

The other significant request was for full IPv6 compliance. The US DoD has recently increased the requirements in the IPv6 arena for RFP's and RFQ's (Request for Pricing/Quotes) to include as mandatory many more IPv6 related standards, many of which are not yet implemented by any operating system, including Linux. As a result, there is a full court press to identify the gaps, distribute the work, and achieve compliance in time for the DoD's required dates.

The remainder of the requirements will be able to be found in the slides from the three presenters, which I'll have a pointer to as soon as they are uploaded, probably onto the Linux Foundation site.

Lightning Talks at Kernel Summit

2007-09-05T08:53:00.000-07:00

Ben Herreschmidt started the lightning talks with the topic of drivers and firmware loading. The idea might be to have two suspend calls, one to enable the preloading of firmware if necessary. A GPF allocation might be blocked during a suspend, possibly holding a semaphore, which is a race that would potentially prevent a resume. Ben has had this happen to him once before. Len believes that this should be using system state, which would potentially avoid the problem. Ben also proposed pointing out a couple of drivers as good examples. Response would be to have a set of example drivers or skeleton drivers. Dave Miller pointed out that beauty is in the eye of the beholder, and others pointed out that hardware is more complex than anyone would like and that makes simple drivers hard to create. Ben redirected the context back to suspend/resume in the context of driver examples.

Dave Miller next took the podium to bring up an issue with 10 GB ethernet driver with Sun's new Neptune chip set. It supports MSIX on the PCI-Express bus. One issue would be that IRQ's could be rebalanced or rerouting all the IRQ's round robin from CPU to CPU. Arjan pointed out that the configuration for IRQ balancing doesn't move ethernet IRQ's around today. Arjan and Dave will talk more about details to make sure the infrastructure reasonably supports 10 GB ethernet capabilities appropriately.

Len Brown brought up hardware sensors and AML as a potential issue. Updating a sensor via AML might affect ACPI, such as converting a sensor from degrees celsius to degrees kelvin, leading to ACPI shutting down the machine assuming it is too hot or too cold. On Windows this doesn't seem to be an issue, but no one knows if it is because they coordinate on conventions, if they have an arbitrator, or if everyone is "just lucky". This was tabled to the mailing list.

Merging i386 and x86_64 architectures

2007-09-05T08:34:00.000-07:00

Andi Kleen and Thomas Gleixner had a healthy debate on the potential merging of the i386 and x86_64 architectures. Linus will accept the patches that merge i386 and x86_64 - he is very in favor of seeing them combined. Comments from the PPC and s390 folks stated that the merge of their architectures was very worthwhile. Andi was still vocally opposed to the merger. There was a long and heated debate both before and after Linus's pronouncement. However, the pronouncement was not changed during any of the resulting debate.

How to encourage Vendors to participate in the kernel community

2007-09-05T07:50:00.000-07:00

Dirk Hohndel started a session to talk about hardware interactions with the kernel. Jon Linville started with the atheros driver and the current open source implementations and the recent kerfluffle about that driver. It looks like that is all resolved now. There are also issues though with concern about government regulations regarding wireless broadcast ranges. While the regulatory requirements are a bit subjective, most vendors are being quite conservative and mostly avoiding working with the community as well. Intel, for instance, is using originally user level and now firmware to help match the interpretation of the regulatory requirements. However, the atheros cards are completely controlled by the host CPU, so the open source/drivers can't use the binary user level or firmware implementations. This is why Atheros currently uses a binary core with their existing driver.

On the nVidia side, the nouveau driver has made some progress but it is very slow going. It is very difficult to figure out the existing registers and semantics for the hardware. There has been great progress with the 2D accelleration model, and working now on dual head operation and laptop operating. There is some hope that Fedora 8 will ship a pre-release of the driver, and it is probably a year from being fully working. There is also the ati500 driver, it is slowly working one card at a time, each PCI ID being enabled one at a time. No work has been done on 3D, although the R500 has the same 3D engine as the R400 which should help.

The other recent player is Via - which asked for a Linux driver. Not sure yet how we are going to work with them.

Chris Schlager, director of the AMD research center. AMD has been successful so far primarily because of Linux. They have recently acquired ATI, which of course happens to make graphics cards. A recently announced project is Fusion, which tries to combine a CPU with a GPU. Of course, the ATI GPU is currently enabled by a binary blob, which of couse is at odds with the kernel community. The current driver can not be open sourced for various reasons. However, AMD will be developing a driver for 2D and 3D for all r500 and r600 devices. All of this will be done in the X.org environment, probably under the MIT license (not yet sorted out but that is the most likely outcome). ATI is also willing to answer questions, although they may not be able to provide specifications.

At this point, Dirk switched to talking about how the community actually makes it harder for vendors (IHV's in this case) to work with the community, which is actually encouraging IHV's to create and distribute binary drivers. Open source participants are effectively getting punished in comparison when working with the development community. Binary driver writers can simply release drivers on their own schedules, with complete control of their own environment. Open source participants on the other hand, work through the unpredictable interactions with the community, and delivering drivers is much more difficult and much less predictable.

The request is twofold - one is to make binary modules substantially more difficult AND to make it much easier to get drivers into mainline. Linus still supports binary modules primarily because they are not derived works of Linux. But, we also need to be better about accepting support for new hardware. The discussion wandered a bit with an emphasis on making it easier to add support for new hardware as opposed to making it harder to run binary modules, both both resonated well with the group.

Andrew Morton on Kernel Quality

2007-09-05T06:15:00.000-07:00

Andrew spent a little time clarifying the existing flow - primarily addressing defects first via email, but then capturing defects wihch are not quickly resolved in bugzilla. Andrew pointed out that many bugs get "lost", some people don't respond to Andrew's email to put their bug into bugzilla. If the bugs are sent to some smaller mailing list, the bug may also get lost. Also, bugs that aren't resolved within a couple of days are typically lost, and there is no way to find and track that defect afterwards. Andrew went through about 3500 lkml messages and found ~50 real-looking bug reports which were not responded to adequately or at all. And, Andrew spends a lot of time nagging people and is getting fairly fed up with it.

Linus believes that most people do not read the kernel mailing list any more. Which could be a problem that makes finding such defects hard to find. The proposal is to consider creating a list just for reporting kernel bugs.

Andrews' wrap up questions were: Should he nag more? Do we need a professional nagger? Why are people ignoring so many bug reports? Somone needs to monitor other mailing lists for unloved bug reports.

Code review is good - an hour of review of someone else's 100 person hour patch can help get that code accepted, increase its overall quality, and improve the quality of the submitter's future patches. But we are not doing enough - many patches in Andrew's inbox get no review or just trivial comments. Also, when Andrew commits a patch, he'll cc a few potential reviewers, which doesn't work too well. And, this policy is only used for patches going through the -mm tree when headed for mainline. Lots of stuff goes from developer, into the subsystem tree, then into mainline. The problem stems from git trees being pulled directly without sufficient review of the patches internally. The suggestion is to send patches for review when you merge them, not when you send them to Linus. That way it is fresher in people's minds and you don't have to do patch rework during the merge window.

Developers generally don't have incentive to review other's work. For instance, they have other stuff to work in, reviewing is hard, a bit dull, and rather thankless, and for many developers, it is not part of their job description.

Andrew pointed out that last year we agreed that subsystem maintainers would send out a summary of what they were putting into the next merge window during each rc period, but no one has actually been doing that. That would be useful.

Andrew proposed a strawman which would require a Reviewed-by: for merge. The merger decies whether it was adequate based on the quality of that review, availability of review notes (a link to email archive in changelog would be great), the identify of the reviewer. The reviewers reputation will accumulate credibility over time. Andrew's goal is to produce an exchange economy in which coders have incentive to review others' work. The consensus is to start using this process gradually and gently to see how it works, potentially making it mandatory if it works out well. Reviewed-by: does not mean "reviewed for whitespace" - but means "reviewed for correctness and content." Linus believes that there are many small patches and this process may be a bit cumbersome for small patches. In that case, Christoph suggests, the subsystem maintainer might be the default reviewer. There is still some debate that this will not work with small patches. There is still a proposal to try this and see how it works.

The Greater Kernel Ecosystem and Documentation

2007-09-05T05:06:00.001-07:00

The next session covered the greater kernel ecosystem, including tings like glibc, udev, the hal layer, etc. Greg KH moderated this panel as well.

An initial assertion is that sysfs is just another system call ABI - sysfs exposes the internals of the kernel. The kernels internals change constantly. There is some substantial expertise in helping to disassociate the kernel internals from sysfs but the general feeling is that this disassociate needs to be more complete so that sysfs interfaces can become a bit more stable. The primary rule is not to assume structure but to assume hierarchy. Then user applications can traverse the hierarchy to determine where the relevent information resides based on an understanding of the hierarchy. libsysfs was a library that no longer exists. The library was written by non-library experts and assumed inappropriate things about libsysfs. The recommendation is to use hal to access sysfs information.

Michael Kerrisk is the man page maintainer for sections 2, 3, 4, and 8. Usually spending a day a week on documenting. In the process he is finding many buggy interfaces. Michael points out that it is hard to design good interfaces, and getting it wrong is painful. User interfaces are difficult and bug-prone, but APIs are forever. There is insufficient review of the interfaces, and the reviews are usually done by implementters/designers ratehr than by the end users (userland programmers). Michael proposed some formal kernel-userland interface development mechanism. This would include a formal signoff and a suite of userland test programs. Dcocumentation should be written by or in collaboration with a kernel developer, and some of the test code should be written by someone other than the developer.

There was a proposal for including the man pages in the kernel source tree. Debated, with the only challenge being about 50 or so systemcalls which have some wrapper in glibc to make the system call user accessible. There is also a desire to include some API testing in the tree as well for system calls and then enforce the API consistency with those tests. Christoph Helwig has volunteered to work with the copious IBM resources available that are working with LTP to pull just the right tests into the mainline git tree for the kernel. There is a strong desire to have better tests, API tests, and system call tests in general, although the community realizes that testing is hard.

Kernel Summit: Mini Summit Readouts

2007-09-05T03:09:00.000-07:00

The first mini-summit readout was from the Linux Power Management by Len Brown. Suspend to RAM is the primary poster child. The most visible is often the video support. Video restore is a bit different in process from, say, Windows, which adds some interesting challenges. Keith Packard is helping with the Intel drivers in this space. ATA might be functional in 2.6.23, although it is not enabled by default. The distros are currently enabling it, however. People are now concerned about how fast suspend-to-ram is, with OLPC seeing +90ms for USB resume (Greg KH just pointed out that that was fixed; Len hopes to hear confirmation from Marcelo), there is also a video sync issue on resume as well as an audio "pop" issue. And, device power management is currently "joined at the hip" with the hibernate implementation. This is not necessarily a good thing. Andrew Morton asked who maintains "suspend to ram", Arjan asked if there was a design for "suspend to ram". Len replied that "the community" maintains suspend to ram, and there was a document describing parts of suspend to ram. However, Len points out that suspend to ram support is very under-resourced today. And Kleen pointed out that in part this is a driver problem since it requires support from every driver which is in use on a given hardware component. A maintainer with a clear vision for how suspend-to-ram would be a great thing.

Suspend2 is now officially TuxOnIce and is out of tree. No plans for it to come back into tree but it is a very popular with some people. It is reportedly a more friendly community for support which might be a reason for the popularity of suspend2. Rafeal is actually doing a fair job of supporting suspend-to-ram today according to Linus. However, getting Radeon drivers to suspend and resume correctly is a crap shoot because of the complexities of the firmware, drivers, etc. Arjan is arguing strongly for some sort of a HOWTO - there is some push back but in general, Arjan's point is to document what is working today and let the

cpufreq: p-states break "idle accounting", fewer governors is better today. Dynamic Power Management (DPM) has been out of tree, although there is no longer general disagreement about the approaches within the DPM community.

Ted did an update on the Filesystems BOF that was held at FAST in February. Most of the session was a read-out of the work that people were doing. About 50% of the attendees were research focused as opposed to Linux developers. Part of the result of that was some direction of the research folks on what the key issues were that would need to be addressed. In this space, there was some progress on the unionfs capabilities. On the downside, there wasn't enough time to really dive into relevent Linux topics. A full write up is available at the Usenix site.

James Bottomley represented the remainder of the IO summit, which included some discussion of problems in fibrechannel (I missed the details), and a discussion of upcoming new technologies such as disk drives fronted by solid state/flash devices. There was some concern about performance and how quickly the flash devices might fail. The disk guys were saying that the disk driver sector interface was preventing the underlying firmware from doing content motion based on hot spots, bad blocks, etc. They were hoping to redesign disk drives to support objects instead of being sector based. This has large impacts on things like RAID, for instance. This would obviously break all elevator algorithms today, since it is impossible for the OS to determine where any two blocks are on disk. The technology is still about 5 years out from a disk-only point of view, although this technology is in RAID drives today. NFSv3 was implemented on a disk drive as a project once, per Alan Cox. There is some expectation that NFSv4 or pNFS will be implemented first on drives, before a pure object model.

Martin Blight provided an update on the VM summit held on Monday. The first observation was Deja Vu - many topics were covered a year ago. Realistic benchmarks are still difficult to find. Hard to get repeatable tests if swapping is involved. Page replication was discussed, as was slab cache de-fragmentation. One continuing proposal was to split dentry inodes from files in the dentry cache (okay, I think I missed something here) but this is always harder in practice than it is in theory. The anti-fragmentation code is now likely to be merged. Larger order page allocations were discussed for several reasons, primarily though for large filesystem blocks on disk. Containers was another large topic, including how applications interact with each other on a single machine. Google (Paul Menage) has done one solution, Balbir has another, which is a bit more complex but is probably a better long term approach. Another topic was the complete removal of ZONE_DMA and adding a similar capability more tied to actual hardware requirements.

Avi Kivity next provided an updated on the virtualization mini-summit. Lguest, LinuxonLinux, KVM, UML, Vmware, Xen, x86 vendors, s390, ppc, ia64 represented as well. The running joke was that long explanations of x86 functionality requested by the s390 people was usually ended with the comment "oh, I understand now, we have an instruction that does that" ;)

The first topic was on performance, and how the hypervisor needs to present NUMA characteristics to guest, realizeing that those characteristics can change at run time. Also, cooperative paging/hinting, e.g. CMMS patch set would be useful. And, the group noted that hardware is advancing, including NPT/EPT which solve the shadow page problem, vmexit time reductions, and several targetted optimizations.

Another topic was on the interfaces for guest/hypervisor communication. The most important one here is that guest/hypervisor communication must be done via physical pages (the guest and hypervisor typically don't share virtual page mappings which could be used for communication).

Paravirt_ops paravirtualize more or less of the guest. All solutions find Time as a common issue and hardware is unable to help with this.

A major thread was that the virtualization solutions need to MERGE - being out of tree makes use of the capabilities and validation of the features nearly impossible.

And then on to lunch! ;)

Linux Kernel Summit 2007, Day 1, Cambridge, UK

2007-09-05T01:08:00.000-07:00

After a pretty uneventful flight (always the best kind) and an easy 2.5 hour bus ride from Heath Row, I'm here and rested. Jonathan will likely have some good coverage for the the mini-summits and linux.conf.eu, and UKUUG which were held here just prior to this year's kernel summit as well.

Ted introduced the summit as the first one outside of North America, and described the schedule as a bit more upleveled than usual. As usual, Ted pointed out that the schedule, location, and content are always open for discussion as the program committee is always trying to make the event as useful as possible.

The first panel was moderated by Greg KH, discussing how the distributors are working with the kernel community, and how they are satisfying end user needs. The first long thread of discussion debated the long cycles for delivering enterprise kernels to end users (2-3 years from feature in mainline to kernel/distribution with related features in the hands of end users, coupled with the fact that most customers test internally for 3-12 months before deploying a kernel). Greg KH proposed updating the linux kernel version more frequently (yearly? twice a year?) within an enterprise distribution. Dave Jones pointed out that just updating the kernel often requires that a number of user level packages also need to be updated, which increases the timeand effort for updating the distribution. The downside of frequent updates is that the amount of kernel regression testing needs to be increased pretty significantly. This might be more testing on Linus' tree, Andrew Morton's -mm tree, etc.

The major theme is how do we get better testing, better review of new code, and ultimately features faster to end users with a higher level of quality and fewer regressions. In other words, more, better, faster. ;) While the enterprise kernels are claimed to be very close to mainline (some key differences from some Enterprise kernels are: Xen, SystemTap, AppArmour, Real Time, utrace, module signing, Novell debugger, some NFS code, etc.), they are unfortunately typically close to an *old* release of mainline, usually 6-18 months older than mainline. And, there are pressures to include capabilities in distributions ahead of mainline for additional vendor and distributor value. These competing pressures - primarily stability and additional features (quickly!) are fundamentally at odds with each other.

Ingo Molnar pointed out that customer upgrades are a very emotional event - the fear and uncertainty of an upgrade balanced against the gratification of new features and capabilities. kABI has some validity as an emotional balance for perception of stability to existing customers.

A sub-thread on hardware platform availability as a distinct problem from some of the additional capabilities offered by distributions. In other words, splitting up new features from new hardware might be an option. But then Greg KH quickly shot that down suggesting that the two problems are too similar to break up and the solution is likely in the same space.

Generally speaking, the problem was recognized, but the only potential solution discussed was to update kernels a bit more frequently in the enterprise version of the distributions, e.g. every 6-12 months or so.

Dave Jones has a list: Myth: Moving to an upstream kernel magically fixes everything. Each Fedora release has about 500 open bugs, about 1500 open bugs for released kernels, those bugs do not match the 1500 bugs in the kernel bugzilla. Some bugs are isolated to bizarre hardware that are hard for most people to pick up. However, many just need a good developer to look into those defects. Very seldom are problems analyzed for root-cause and people often ask for people to "retry with the latest release" just to see if the problem magically went away. For instance, 2.6.22-rc5 doesn't even boot on Dave's laptop, e.g. a regression. SATA tends to be especially bad right now. Suspend/resume works/fails on an almost alternating basis. Laptops are sufficiently distinct that a fix for one laptop often breaks another loser.

General comments centered around the fact that there is not enough focus on defects in general in the kernel community.

Dave Jones has a list: Myth: Moving to an upstream kernel magically fixes everything. Each Fedora release has about 500 open bugs, about 1500 open bugs for released kernels, those bugs do not match the 1500 bugs in the kernel bugzilla. Some bugs are isolated to bizarre hardware that are hard for most people to pick up. However, many just need a good developer to look into those defects. Very seldom are problems analyzed for root-cause and people often ask for people to "retry with the latest release" just to see if the problem magically went away. For instance, 2.6.22-rc5 doesn't even boot on Dave's laptop, e.g. a regression. SATA tends to be especially bad right now. Suspend/resume works/fails on an almost alternating basis. Laptops are sufficiently distinct that a fix for one laptop often breaks another loser.

Debian - drivers is the biggest problem. The number of ethernet, wireless, video cam drivers have a much wider variety of drivers which are not accepted in mainline, unionfs, squashfs (maintainer was scared off)

Linus strongly advocates getting even crappy drivers into the mainline tree as early as possible. Alan Cox was pointing out how buggy that can make the distribution. Linus' point centers around the fact that the code is much more public if it is in the tree, Alan's point is that
the code is never going to get cleaned up and will harm everyone else in the process. Linus is focused on those that are going into distributions *anyway*, so why not get them into mainline. And, the community then has a better chance of fixing them. Wireless drivers should be in mainline; they are sitting in Dave Miller's tree at the moment.

Deepak representing the embedded side doesn't want to put drivers right into kernel.org - because in many cases the code is written for multiple OS's and is so far from kernel coding standards that it is almost completely undebuggable. While his point was recognized, Linus reiterated that getting things into mainline is the best way to get the code cleaned up, and Greg KH reiterated that there are 85 kernel developers just waiting to help maintain drivers in mainline. One of the key problems is that many drivers go with either very, very old hardware or very new hardware, most of which is not available to most people. Greg KH has offered to help with any drivers that are out of tree and need to make it into the kernel.

Microsoft has many tricks...

2007-08-29T12:02:00.000-07:00

It is interesting to watch as Microsoft tries different strategies in dealing with the open source community, open standards and interoperability. This interesting ballot box management trick with OOXML is clever (oh, and if it is so good, repeat it, over and over again). Basically, manage the voting populace to ensure that only your voters turn out for a vote.

So, does that imply that they know that their proposal is inferior (oh, with 6,546 pages, I guess they didn't leave much undefined, compared to ODF's mere 867 pages, right)? Does it leave gaps which allow them to maintain a proprietary interface and extensions, and therefore lock out open source or 3rd party players? Does it add obscurity instead of clarity, allowing only mega-monolithic companies to play in the open document specification? I like Google's stance on this, which is well thought out. And, in watching the discourse as the Linux Foundation pulled their response together (and some comments from the open source desktop architects), the focus was not on Microsoft vs. Linux (which at some level this *is* about, although by no means the primary focus) but instead the architects clearly focused on the technical aspects of the proposals.

I'm not impressed by the style of tactic here. Microsoft should have enough technical savvy that they should play this game openly and based on the merits of any standard. While I'm sure the days of corporate jury rigging of standards for vendor advantage are nowhere near over, any standard relating to true opennes affecting so many end users should get much better treatment and input that favors we end users.

A laptop in every (airline) seat

2007-08-24T09:22:00.000-07:00

Now this is cool! Singapore Airlines is putting a Linux laptop in every seat. I had the opportunity to use my laptop with wireless on Luftansa flights via Boeing's Connexion service referenced here and that *really* made my life useful. Now if they combined this with a Xen base for that Red Hat version, you could install your own OS and images on a USB key or a USB disk, boot your own environment, and be up and on your way.

Now where does Singapore Airlines fly? I have to figure out my next boondoggle! ;)

The Sun begins to Set, but is Java on the rise?

2007-08-24T08:57:00.000-07:00

As my colleage Paul said, you just can't make this stuff up. Jonathan Schwarts blogged that they are changing the Sun Microsystems stock symbol from SUNW to JAVA. Isn't that about as smart as changing IBM's ticker symbol to, oh LINUX, or maybe AIX? Most big companies that are successful build a rounded portfolio; to emphasize something that is so close to being a commodity as the rallying call for an entire company suggests that perhaps there isn't enough sunlight getting into the corporate offices at Sun any more. As a company which once set the standard for workstations, networking, networking file systems, etc., I think this is a pretty silly move and sounds more like desperation. This seems more like a child yelling "look at me! look what I did!" than a company with a clear vision of the future and their part in it.

Novell owns the UNIX and UnixWare copyrights!

2007-08-10T21:52:00.000-07:00

Groklaw has this cool article up which shows basically the conclusion of one of those big SCO cases. Now, the question I've had for a very long time is: If Novell owns the copyright, could they consider releasing the source to all of the Unixes and Unixware that they own under, oh, BSD, GPLv2, GPLv3, something interesting and shareable? I, of course, would vote for GPLv2 in case there's any compatibility code that Linux could borrow, but BSD might be mostly as good. That might be a bit like pouring an ocean of salt in a very large wound (or would that be a lake of salt, hmm?) but it might be a generous gesture on the part of Novell to balance out the Microsoft Kum Ba Yah refrain of earlier this year.

Ah, I can wish, I guess. It probably doesn't matter all that much in the end but it would be an interesting symbolic gesture.

Next Generation Data Center

2007-08-10T14:59:00.000-07:00

I talked to a few customers at the Next Generation Data Center expo who had one very interesting observation. While there is a lot of talk today about the the design of the next generation data center, how it optimizes space, how it handles cooling, how it has a lower power consumption profile (or adapts power consumption to load), how it minimizes heat output, how water cooled heat exchangers reduce cooling costs, etc., there are no good templates, recommendations, or guidelines. IBM Global Business Services does provide a contract rate for designing data centers with most of these in mind, based on your lab size, your power constraints, your cooling constraints, etc., but everything is still a one-off design and there are still a lot of tools missing along the way.

It *is* clear, however, that space, power, and cooling are still primary concerns, and the normal constraints of price and performance are not released very much if at all. There are little things coming that will help with this, such as powertop which will help optimize applications and workloads a bit for power consumption. However, that is only a drop in the bucket compared to optimizing data centers for power consumption or cooling. There are some mechanisms in use with workload consolidation such as IBM's recent announcement with System Z which reduce costs in terms of power, cooling, systems administration, while increasing application availability and throughput monitoring.

However, it is starting to look like the holy grail for large data centers involves a combination of software metric measurements of power consumption and heat generation, combined with the software to move workloads around, reduce power while meeting service level agreements, and powering machines or portions of machines on and off on demand. Most of that software does not exist today, and what does exist isn't yet targetted at managing a full data center that many corporations use today. I believe this will be a key area to watch over the next year or three as more and more companies get on the Green bandwagon.

rPath and Software Appliances

2007-08-06T13:56:00.000-07:00

My chosen after lunch dessert at NGDC is another virtualization topic, this time presented by rPath. rPath provides a means for making software appliances and being able to distribute them. Software appliances attempt to address one of the growing problems in Linux which is the application certification issue. Specifically, a key customer problem today is that applications are often certified on a subset of all of the Linux distributions that are available. And, what is worse, is that different ISV's certify their applications on different distros, but customers expect to buy any application off the shelf and run on their chosen distribution. Alas, this is becoming less and less the case. I've been at a number of customer sites where the number one complaint was that their key applications were not certified against their chosen RHEL or SLES distribution. Or worse, some were running Debian, Ubuntu, or some other distro. At the core, the distros are very similar, and most applications will *probably* run. But, if they don't, the customer gets to keep all the broken pieces.

So, how do appliances help with this problem? Well, simply put, the encapsulate all of the key attributes, including potentially the operating system!, into a single image which can easily be distributed, loaded, and delivered to a system with a hypervisor, such as VMware ESX, Xen, etc.

Examples of appliances can be found on the rPath site, like this LAMP stack, for instance.

Erik points out a few problems with virtual machines, some of which are noted in the previous blog entry. This include the fact that proliferating virtual machines are just as painful to manage as physical machines, and managing things like security updates take the same amount of effort as managing physical machines. Also, hypervisors today have some dependencies on their operating systems if they are going to get the benefits of the leading edge para virtualization techniques and such. While those compatibility issues should fade away over time, that is at least a short term consideration. As a result, a virtual image's kernel may be build for interfaces to a given hypervisor (Xen is probably the worst in this space right now, but mainline kernel API's for domU guests went into mainline just recently and should start to show up in distros over the next year or so).

On the flip side, Erik pointed out how well Software Appliances fit in with the Software as a Service (SaaS) model that is becoming so popular. In short, it fits in very nicely - a software appliance is basically a software service. There are a lot of other advantages, such as the fact that a software appliance is easier to test (it is always the same stack, no matter where you install it), easier to support (stack is well known, all customers using the same appliance), and easier for teams to configure (again, all components are the same). Erik believe that Software Appliances are actually better than SaaS for a few simple reasons: there's no need to worry about multiple applications residing on the same OS - apps are isolated by a virtualization boundary; there is no internet latency - multiple apps/software appliances can reside on the same physical system; no remote/unsecured data; no significant data center infrastructure - software appliances are virtual.

Erik provided a pretty common sense list of best practices for creating software appliances that I won't re-iterate here but the guidelines are definitely useful - small, simple, easy to administer (no CLI, minimal to no configuration required), etc.

Something that Erik pointed out that I have not really looked at is Amazon's elastic compute cloud. It has the ability to auto-create and host software appliances. The rate is a mere ten cents an hour in US Dollars. He demonstrated the creation of a MediaWiki site in less than ten minutes from click, create, configure, use, system updates all based on a preconfigured software appliances. Definitely a pretty cool option.

In short, I think software appliances will be a major boon not just for the SMB market but even for large corporations who need to deploy anything from sandboxes to entire racks of new machines with their custom workload. Since most of these solutions require a virtualization layer/hypervisor in here somewhere, I believe this will provide a significant push for virtualization in leading edge datacenters as well.

Virtualization tutorial at Next Generation Data Center conference

2007-08-06T09:30:00.000-07:00

I'm in San Francisco's Moscone center on a chill and wet day in San Fran (what, did I really expect warmer weather than Oregon's current cold snap?). I'm sitting in a tutorial from Dan Olds on Virtualization at the Next Generation Data Center expo. Yes, this has been an area of interest of mine for quite a while.

Dan started with some rather (well, to me, anyway - there is a ton of info like this floating around the net already ;-) boring comparisons of workload/machine utilization and the benefits of consolidation. What was interesting was a number of surveys that his company engaged in which showed that traditional Unix customers have been more proactive at embracing virtualization than the current x86/x86-64 class of customers. And, while comparing Unix and x86-64 has initially been a bit confusing, it is clear that he is looking at the traditional strengths of Unix systems on non-x86 platforms and their built in relationships with their OS & Hardware to support virtualization, as compared to the x86-only virtualization solutions enabling multiple OSes to run on x86 (e.g. Windows, Linux, Solaris x86). But the trend for virtualization is on the uptick in the x86 space, with still a sizeable number of customers not convinced of its overall value.

Dan talked a bit about the the reasons why some customers don't see the value on virtualization, primarily because single rack mount servers are often so cheap, so capable, that purchasing and deploying a single small server is cheap and easy. It isn't until a site starts running into space constraints, power restraints, or cooling constraints that the incremental deployment of small servers becomes problematic. And, most people don't monitor the utilization of those small machines because they are not viewed as precious resources. Ergo, lots of potential waste cpu bandwidth, power consumption, heat generation, etc. Also, the number of sysadmins increases quietly as the number of servers increase, especially if the servers are actively managed for latest security patches, application updates, etc.

Dan talked briefly about consolidation from rack mount to blade servers and didn't see that as a major savings in anything other than space, and *maybe* over time power/cooling. Most people using blades aren't really doing any virtualization today and only improving on the footprint part of the consolidation story, where there is much more to gain when reducing the number of operating system instances, hardware platforms. Later Dan talks about making sure to measure *all* of the cost savings opportunities for virtualization which I believe is a critical component of any successful virtualization deployment as it helps understand exactly why virtualization is so important.

Dan next spent some time comparing VMware ESX and Containers solutions. He talked a bit about platform virtualization versus containers. He described how both were valid and often necessary, depending on your type of comparison & consolidation. VMware (or Xen) provides full OS isolation, containers does not. Containers provides low overhead therefore better performance. VMware allows multiple OS's, multiple versions, Containers is limited to a single OS, single version. VMware instances are each management independently, with Containers there is only a single OS to maintain, which may reduce systems management overhead. Containers is relatively a lot cheaper than VMware per socket.

A question from the audience was: why do we need virtualization when Unix/Linux today scales well and supports multiple applications on a single system? Answers vary across the board, from application isolation, security concerns (full security isolation between virtual OSes), simplicity of management of applications, ability to measure impacts and utilization of a specific workload, and, over time, the ability to migrate applications from one system to another or from one virtual environment to another.

Dan spent a fair bit of time talking about how to build a plan for moving towards a virtualized environment. I'll skip the details but most of his approach was strategic in nature rather than tactical, and he missed out on a lot of the process and efficiencies that can be brought to bear while working more on the justification and engagement of management and executives into any such plan.

Net summary: Virtualization is an ongoing trend, starting to become more prevalent in the x86 environment after being primarily reserved to high end mainframes or higher end Unix systems over the past few years. There are many benefits to be gained from virtualization, not all discussed here, but including consolidation, energy savings, simplification of management, and general simplicity of deployment. This talk left out the benefits of rapid prototyping and rapid deployment of solutions, for example. But for beginners this was fairly useful, I believe.

OLS Notes, Xen on KVM Talk

2007-07-20T15:42:00.000-07:00

So, unfortunately (or wait, maybe that's the point of a good conference!), I spent a lot of time networking and multitasking during the conference, in between sessions and sometimes during sessions. As a result, getting my notes cleaned up and posted didn't happen in real time quite like I hoped. But, there are still a few cool things worth noting from the sessions that I attended.

For instance, Ryan Harper did an interesting talk about how to run Xen guests on top of KVM. This is especially cool because it manages to avoid the core of the Xen vs. KVM debates and allows guests to run on either environment. So, test and compare, pick your favorite, and the
same guests will run.

Ryan spent a bit of time talking about difficulties in working with Xen which made KVM a bit more attractive at least for Xen/guest developers. Specifically, rebooting the hypervisor on changes was a bit painful - all guests must obviously be started (hey, where is guest checkpointing? ;-). And, whenever Linux or Xen changed, you often have to rebuild and reboot both still. You also need to occasionally reboot the user space daemons when running Xen, probably based on changes in Xen or the client.

From an end user perspective, there are also a number of existing issues, including a reboot to install Xen the first time (or on some updates), some issues where the paravirt ops for guests are still not mature and not yet fully interoperable. And, from the Distro perspective, the two
primary distros are on different snapshot timelines, and all of the smaller distros are obviously running with a huge variety of release dates, stability goes, paravirt awareness, etc. So, the dream of running any guest on any distro + Xen or any Xen release without problems has not
quite materialized yet. And, the installation and management tools for guests or the DOM0 are not consistent across the board. And, there is not currently (maybe some day?) a set of paravirt ops for the Xen domain zero (DOM0). This means that the current DOM0 build is fairly incestuous with the Xen hypervisor and thus the two often need to be built in lockstep (the guest paravirt ops is moving forward reasonably well and could be in Linux 2.6.23).

So, if Xen has all these hangups, why even use it and not switch entirely to KVM? Well, Lguest and KVM require hardware virtualization, for one. Second, Xen is already pervasive in Red Hat and SUSE releases as well as existing appliances, and there are a few interfaces which still perform poorly in KVM, such as Direct Paging. Xen also has ports which enable Solaris and Net/FreeBSD to run as Domain 0 for those that want that sort of thing. So far, KVM is completely Linux specific for the (roughly) equivalant domain 0 activities.

KVM does have some significant advantages though. For instance, there is a huge body of re-usable code such as device drivers and hardware enablement. KVM is already upstream in mainline Linux. It is integrated with QEMU for full hardware emulation/virtualzation. And, it has well established interfaces for the user and for userspace/ kernel interactions.

Working with Xen guests on KVM also helps expand the virtualization community (note, this is like the forking that James Bottomley talked about in his keynote speech at the end of the conference, and adds value in exactly the ways that he described there). It also helps to expand the KVM community to systems without hardware support. Ryan expects that this approach will enable virtualization on an as-needed approach, simply by creating loadable modules for
guest, kvm, xen, and potentially even various versions of those such as Xen3.0.3, xen3.1.

His work focused on some key design points such as keeping the monitor small and simple, no hypercalls, shadow page tables in the monitor, and using existing Linux capabilities whenever possible. He also did not add requirements to run Xen daemons on the host - those can be hidden in KVM or QEMU.

A final note on KVM - it is not a replacement for Xen deployments, use dedicated Xen for top performance.

Ottawa Linux Symposium, Day #1

2007-06-28T09:31:00.000-07:00

Day #1 at OLS started off nicesly with Jonathan Corbet and his "State of the Linux kernel" talk. I just recently also saw this talk given at the Linux collaboration Summit. Jonathan updated the slides a bit but also included pointers to sessions related to key technologies making their way into the Linux kernel.

I next sat in on Paul Menage's talk on containers. While Paul is clearly a sharp guy, unfortunately, the big picture of his talk and goals weren't super clear in his talk. Paul has the leading implementation for something called containers, but somewhat unfortunately at the momemt, there are several things known as containers. He made an attempt to show that he was integrated with all of them by the end of his talk, although, I think the level of integration with some of them is pretty weak. I think the code and paper are probably better referencdes than his slides & talk.

The next session I attended was near and dear to my heart - I'm a big fan of appliances as the next wave of deployable solutions. However the work that David Lutterkort in Red Hat's Emerging Technologies group was doing was new to me. Of course, being on the program committee, I did have *some* preview of his high level goals, but the tools, such as puppet, that he's working on where completely new to me. He also mentioned Kronolith which I have yet to research. But the talk was good and the technologies are definitely cool for rapid deployments, pre-tested configurations, and easy of management for systems administrators deploying
similar workloads on lots of similar machines.

More to come...

LIVE! from the Ottawa Linux Symposium!

2007-06-27T16:51:00.000-07:00

Well, it would be live if my wireless cards worked better with Linux. I might be able to fix that. My primary wireless card is an Atheros and I hate it with a passion. It has sucked more useful life from my body than anything else in my standard operating environment (other key examples were early uses of apt-get upgrade on debian, and being an early adopter of Fedora 7).

Anyway, the opening/greeting/mingling location was at a bar called Vineyards in the Byward Market area and, whileonly a small percentage of the registered crowd showed up, the place was
very busy and the number of actual code developers and contributors was quite high. One of the concens going into this year is that the move of the Linux Kernel Summit to September in Cambridge UK would pull a lot of the core developers away and reduce the quality of the
networking opportunities at OLS.

However, the "newer" model of holding key mini-summits has substantially mitigated that issue, if, indeed it would have been an issue. There was a virtualization mini-summit, a power management mini-summit and, I believe, a third mini-summit that were all held prior to OLS. Those mini-summits pulled in deep experts in those areas, and several other development areas were well represented, including ext4 developers, various architecture maintainers & core developers, dm/md developers, storage folks, container folks,

Word from the organizers is that in excess of 660-670 people had registered for OLS this year, with a few stragglers still signing up at the last minute. There seemed to be a larger international component, with more people from Japen that previously, as well as a strong
European contingent.

There appeared to be a large amount of beer and a fair bit of wine consumed at the bar that evening, which seemed to enable a free flow of information and plenty of mingling. I had a chance to talk to people about everything from power management, containers & networking (Dave Miller hates container namespaces!), device mapper and support for EMC multipath hardware, a rework of the Linux sound subsystem, and a number of debates in several of those spaces, just to name a few The night ended a bit early for some when the bar started closing down a full 45 minutes before Ottawa's otherwise early 1 AM bar closure. However, all for the better since a 10 AM start was in the works for Wednesday morning.

Late note: these blogs may be more "as it happens" tomorrow since I think I figure out how to get the latest MadWifi and linux-2.6.22-rc6 to work just a bit ago!

End Users and Linux: What Works, What Doesn't?

2007-06-13T16:20:00.001-07:00

Moderator: Tim Golden, Bank of America

Jeremy Allison, Google
Chris M
Randy Terbush, CTO of Infrastructure Architecture and Strategy at ADP

What factors led your company to adopt open source?

Randy: Started pressing at ADP about 4 years ago, wanting to provide a common abstraction above all hardware interfaces, driving towards a more utility computing model.

Jeremy: Google gets complete control of their environment. Jeremy likened it to Dr. Who's Tardis - weirder on the inside than the outside.

Chris: Has consulted for around 30 open source companies. Most started with the fundamental belief that open source was the better methodology in a philosophical sense.

Tim: Initially, for BofA - it was about the Money. In the core infrastructure space money was the initial issue although after a bit that changed to a more functional set of concerns. One key is a faster way to bring products to market.

Question: What does Linux have to do to win against other platforms when your company considers new deployments?

Randy: Constantly competing with knowledge and awareness around Solaris, and ensuring that Linux accellerates rapidly enough to compare well against Solaris.

Chris: Linux helps ensure that there is no single vendor lock in in the core portion of the stack for a number of European companies that Chris consults for.

Jeremy: Google is obviously highly Linux, but engineers can choose Linux, Windows, or a Mac box. Google uses Linux "when it makes sense" - it isn't a religious decision.

Tim: Maintaining a strong Linux presence on site is an ongoing challenge. There are constant competitive pressures to consider alternatives and repetitive TCO analyses are common.

Chris: Linux often enters "like a water leak from below" and rises until it gets some visibility - and that visibility is often initially negative from senior folks who are not familiar with Linux. Chris also pointed out the dynamics of departments who have a human desire to grow (or at least remain stable) and Linux plays against that desire by reducing costs - and reducing costs (being successful) implies that you can operate with less budget in most IT shops. This is not exclusive to Linux but a standard trend in deployment of platforms that increase or improve efficiency.

Jeremy: provided an anecdote of selling servers with Samba as a way to combat that thinking - $600 per server (for open source Samba!) helped make people think they were getting monetary value from their cost reductions (using open source).

Question: Why do you like open source?

Jeremy: "Control of our Destiny"

Randy: It's about standardization. They aren't really seeing cost benefit, but the real value is in standardization across the board. Sun is basically giving away Solaris as well, so on a price comparison Linux has no strong lead. Commonality of OS is a real key.

Tim: Went after cost savings, with a full, multi-year TCO assessment to show that value. There is probably a stack of benefits in moving to Linux. Enabling Linux helped bring in a bunch of other open source projects since it paved the way in terms of risk and license assessment, general comfort with open source, etc.

Question: What are the biggest challenges in deploying Linux within your organization?

Tim: This community has been a challenge. Is it a "big C" or "little c" community? Differences of opinion often lead to isoluation of communities and seperation or ostracizing a member of that Community. [The debate between whether various people are part of "the community" has come up a couple of times during the day --gerrit].

Jeremy: in talking with Andrew Morton about this question, the answer seemed to be "The engineers keep fiddling with it!" This winds up being a pain point because things are constantly changing, and not always for a good reason.

Question: If you were Linux God for a day what would you do?

Tim: Make a full ecosystem around SystemTap

jeremy: Make all distros release the same Linux kernel and glibc across the board.

Chris: Make the power management stuff work better. Greg KH: "It works for us." If it doesn't work for you, let us know, send us your laptop and we'll fix it for you. There are two ACPI implementations - one for Linux, one for Windows. ACPI is not power management and not suspend/resume.

Randy: Would like to see a single, unified, certified Linux distribution. Clarified: Ideally, a single source from which Red Hat and SUSE would pull from, where the single source was was the certification point for applications.

Greg KH: One Linux Distro, the Uber Distro: There some very large

Jeremy argues that the LSB is the real answer to that. [I don't believe that - LSB is such a teensy portion of a full distribution that it isn't practical to make an LSB large enough to cover the real issues, although LSB is a good step in the right direction.]

Question: As an end user, are you part of the Linux community?

Randy: I think we are - in part because we support the community with dollars.

Chris: Most companies employ individual contributors today - there are very few programmers living in their parents basement. As such, most companies are in some sense part of the community.

Jeremy: Yes, there are some

James Bottomley: The Linux Foundation now has a mechanism for accepting dollars to help direct those funds towards programs, coding, projects, etc. Projects can be proposed to the Linux Foundation as well for development and focus.

Powered by ScribeFire.

The Legal Protection of Linux - Patents and Licensing, GPLv3 and the Future

2007-06-13T15:12:00.001-07:00

Moderator: Andrew Updegrove, Partner, Gesmer Updegrove

Mark Radcliffe, Partner; DLA Piper US, LLP
Jason Wacha, Vice President, Corp. Affairs; General Counsel, MontaVista Software
Karen Copenhaver, Partner; Choate, Hall & Stewart
McCoy Smith, Senior Attorney, Intel Corporation
Abdy Raissinia, IBM

Mark Radcliffe:  Has been practicing for over 25 years.  Has 40 lawyers working in Intellectual Property issues.  Ran Committee C for GPLv3.  Impressed by the flexibility of the group crafting the GPLv3 and converting the license into something that people will actually use and accept.  This is not the way that licensing is typically done!

Jason:  working with large companies on using Open Source on hand held devices, etc. including building their knowledge and comfort of working with open source, open source licensing and intellectual property concerns.

Abdy:  Referenced OIN, patent attorney for IBM Linux Technology Center

Andy Upgrove:  Worked with the X Consortium ages ago.

Karen:  First ran into GPL in 1994 on a library.  Found it to be "really werid".  Probably too weird.  She asked the developer to use something else, anything else.  The developer over time pointed out that the math library was so highly examined (open source participation) that there was no other viable option.  Karen re-examined and found the GPL to be "really weird".  She then spent some quality time with the license as her first foray into open source and open source licensing.

Question:  How might open source and open standards play together in the future?

Jason:  this will be pushed by commercial forces.  Linux started as professionals playing as amateurs, but now people are getting paid as professionals to work on Linux.  That commercial force will drive the future directions and leverage the deep & broad aspects of the operating system.  Andrew: Do more developers need to be involved in the process?  Should the community be more involved and activist in the standards (and licensing?) process?  Jason:  the community is less clear since it includes individual developers, commercial entities and many others.  Mark:  as more applications and environments mix open source and proprietary applications and uses in a heterogenous environment, their will be more focus on licensing and related issues.

Mark/Andrew:  In the US, corporations are pushing open source, in Europe, developers/linux community are the primary force, in Asia, the government is the primary force.  Addition:  places like Brasil also have a very strong government support situation, where in the US other proprietary concerns are preventing a strong US government push toward open source.

Jeff Liquea - works for LF, also part of debian (which is well known for "amateur laywering"  - speckling of laughter) - question:  what do you think of amateurs writing licenses, giving interpretations of licenses, etc.?

Karen:  Lawyers probably wouldn't have had the creative capacity to have this vision - without those amateurs, at some level the open source movement wouldn't exist.  Lawyers are now trying to help mold the existing work into something that has a solid legal foundation.

McCoy:  This question came up in part during GPLv3, amateurs may not truly capture the goals that they want to achieve.  However, communities have an understanding of their guidelines and are self-policing to some level, which is a good thing.  Of course, there is some potential for problems down the road when amateurs are driving.

Abdy:  More and more, attorneys are actively reviewing these licenses and there is more overview in a legal sense of the new directions.

Jason:  "amateurs" or the developers who have a vision for what they want to do and can set the boundaries work well with lawyers or attorneys to help craft better licensing.

Karen worked with a mock court with Federal Circuit Court Judges as a way to evaluate some of these licenses, and the effort was definitely challenging.  The judges frequently referred back to the definitions of "derived works" because the area is very complex.  There is a definite concern that a bad court case could set a bad precedent.

Mark:  as the amount of revenue pressure from Open Source on proprietary companies increases, the amount of pressure on open source licenses and potential legal challenges is likely to rise.

General comment:  Several pointed out that lawyers mostly don't understand open source and related licenses, and even the panelists took a while to embrace it.

Bernard Golden:  What is the likely uptake of GPLv3?

Karen:  Things look much improved and there looks to be some significant potential for uptake.

Mark:  the user level community is likely to accept it more readily than some communities, and the VC group looks at the license as pretty reasonable based on his experience with VCs.

Karen:  Clearly the uptake will be gradual and thoughtful over time as well.

Mark:  A number of ISVs are going to find GPLv3 more palatable than GPLv2 in many cases.

Abdy:  The process of evolving GPLv2 to GPLv3 and the presence of many large vendors in that upgrade process strongly suggests that open source and related licensing is here for the long term.

How Do We Get More Apps on Linux?

2007-06-13T14:19:00.001-07:00

Dan Kohn started with the question:  Is Web 2.0 really important?  Isn't a browser on your desktop good enough?

Web 2.0 should solve a bunch of low hanigng fruit, but phones are not yet addressed in that environment.

Legacy code won't be reengineered as Web 2.0 apps - but many have been ported to Linux already.

Kay Tate - we have seen a lot of legacy apps being ported, but more so the small and medium sized business (SMB) are porting to Linux.

Web 2.0 should solve a bunch of problems but clearly won't be the be-all and end-all for software.  The further you get away from the silicon valley VC environment, the less powerful the Web 2.0 message is.  What will inspire people to port more easily to Linux?  A premise in the question is that Eclipse is part of an answer.

Adobe does have web based version of Adobe Photoshop.

Question:  why does Windows Vista have 3400+ certified apps?

Brian Aker:  Many of those apps are clones of programs which deal with zip files.  Linux has one [actually, I think Linux has as many as Windows, most likely ;-) --gerrit].  New applications should be encouraged to port to Linux primarily.

#2:  Yes, new applications should be ported to Linux first when possible.  Customers look first to solve a business problem, then looks for applications which solve that problem.  We need to focus on new applications which fill the current business needs.

Kay:  Distros are well certified with LSB, etc. so that is where we are doing well.

Mike: How do we get more *applications* that run on Linux - not just programs and products, but focusing more on users like bank tellers and the applications that they need.  We are sort of in the space of crossing the chasm - the easy, early adopters have been convinced, the next step is to cross the chasm.  Using gcc & gdb is obviously not sufficient, so we need a strategy, philosophy, culture change to convince people to change.  We need to provide developers with an environment that provides a full development environment - this is a place where Microsoft has done well.  Eclipse helps in this space although Eclipse is not as ubiquitous today in its adoption as any Microsoft development environment.

Ed  Real Player is small and available, tools are relatively easy and convenient to create media that plays in this environment.  Application writers what to make sure that their applications are ubuiquitous - and the Linux market is still relateively small compared to the Microsoft environment.  Having an application environment like the LSB should help address an ISV concern by showing that their application, once built, will run on many distributions, thus increasing the ubiquity of the application market.

Question:  If I were an ISV who has decided to target Linux, what problems would be challenging to address?

LSB helped consolidate the development environment, so it is good.  One of the challenges is the many installers on the various distributions.  [Dan points out that the LSB has been around for 8 years but the number of applications certified under LSB is still relatively small].

Flash is supported on lots of platforms, now certified on RHEL3/RHEL4, Novell and one or two others, and do minimal acceptance testing on a few other distributions.  However, the support matrix is huge already with old versions of Windows, etc., so adding lots of Linux distributions is hard and expensive today.

Mike:  He goes back to the economics of the problem.  Specifically, the number of Eclipse downloads have been 80% windows, 14% Linux, 3% Mac and hasn't changed in three years.  One good factor is that the distros now include Eclipse, which might mean that the number of Linux uses in the field is actually larger.  So the global question is:  has anyone made any money with LSB applications on Linux?  So what is the ROI model for software which provides incentive for people to port to Linux.

Kay: Most of the ISV's that IBM works with are already on Linux.  However, for those that aren't, being able to explain to them that a single source will run on multiple platforms, the source base will be more stable, and they'll invest less across the board, primarily because of the LSB and IBM's chiphopper program.

Darren:  Pointing people to the LSB site has helped people understand how to simplify their porting environment.  Most people tend to port to one or two distros and the remainder of the distros get left out in the cold.

Kay:  thinks the market has gotten better, but Brian disagrees, primarily because of the differences of kernels in each of the distributions.  The differences between distributions are just enough to cause real bugs in MySQL lately, and things have gotten worse lately.  As an example, linuxthreads is still an ongoing nightmare for MySQL.  Tools market is not a lucrative market, so implementing new tools is not a reasonable business model.

LSB application toolkit manager, version 0.1 was just released, which provides a wrapper and front end to the test infrastructure that has existed for a while.  This should simplify application certification/testing.

Cadence SW - apt and yum are a pain for their software - they are having problems with defect support on the variety of kernels and environments - it is too expensive to support their customers on anything other than a small set of platforms.  So, requested that Eclipse help in automatically building applications that are LSB compliant.  The observation is that Linux is not *one* distribution but 500 different distributions where each one expects to be different and yet expects applications to run.  Cadence is not as interested in the desktop but more interested in the server side and con

Dan Kegel - Picasso would like to be added to the distribution but it is a binary only application.  It would be nice if there were an xdg add-trusted-repository [I have no idea what this means, so I may have transcribed incorrectly --gerrit].  Dan would love to see that Wine improved to the point that most applications could run in wine without a port, yet potentially running slowly.  Then market numbers would help demonstrate the need for a native Linux port.

Gordon Hopper from Motorola:  loves apt and yum, having all these utilities in apt and yum seems to be a bad idea since it excludes all of the binary applications.  But having more of the open source applications become LSB compliant and shake the bugs out of the LSB tools and such would be good. 

Pacific NW startup, Linux ISV - as a startup they are required to make money and thus be profitable.  Answer:  LSB is really for the middle tier - big ISV's throw bodies at the problem and solve it however.  However, the LSB is more likely to address the SMB market.  Application was built on .NET originally, then ported to Mono.  The business model for pure Linux ISV's is hard - the comment made was "it sucks to be you!".    The only two technologies that work that way are .NET/Mono, rcp, wine, (and one other was mentioned by the audience).  The ISV model enabling applications first on Mono and porting to .NET seems to be viable.

Anyway, my summary would be that this session had lots of questions and discussion, but fewer answers than most might have hoped.  Hopefully, LSB helps, but it clearly doesn't solve everything.

Mark Shuttleworth Keynote at Linux Collaboration Summit

2007-06-13T11:44:00.001-07:00

I'll skip Mark's intro - he's easy to google (Thawte, Verisign, Space Shuttle, Ubuntu).  He gets the keynote address at the LF Collaboration Summit.

How to collaborate.  Non-traditional folks are starting to hear about Open Source software.  But headlines come from "contests of will" such as Red Hat vs. Novell, MS vs. Ubuntu, etc.  However, innovation works better when people are inspired, when they have an itch to scratch, and you provide an environment for them to scratch their itch.

Community requsts:  pass stuff upstream.  Distro:  we didn't know where to send it (not all projects are well maintained).

Loudest ideas tend to bubble up, look to talk on Poisonous people, how to make sure best ideas win, as opposed to loudest ideas.  We have lots of collaboration tools - wikis, mailing lists, etc, which focus on collaboration within a project.  However, collaboration between projects is weaker.  Translations rarely move upstream.  Can distros create a "star topology" for passing fixes between distros, thus facilitating upstream adoption.

(Battery dying here, time to swtich).

Back now - Mark has been talking about Federating our interactions, with a focus on several facts, including multiple distributions working on the same projects, with similar inputs/bugs/needs, developers working cross projects, e.g. accessibility and translation being places where non-experts contribute to many projects, so many projects on different schedules being collated into distributions, bugs spanning multiple products for a single user-experience flaw, etc.  Mark used the example that creating an account on every bugzilla, sourceforge site, etc. would be prohibitive for any individual who just wanted to contribute translations, for instance.  The model today dos not span multiple projects, although the collaboration models work quite well within individual projects.

Ubuntu used Mozilla and Gnome as models (Mozilla - roadmapping; Gnome, committment to release cycles) as influences.  Each feature develops a micro-community, a branch, etc. in some projects which seems to work well.  IETF has a similar model for creating an idea and building a consensus around that idea.  How do we generally lower the barriers around development and contribution?  Why can't people do an apt-get of the source of a project, create a branch, allowing everyone to develop like the core developers, being able to easily contribute changes back to where they can be reviewed, integrated, etc.

Ubuntu uses bazaar (a brach of arch) for source control.  Matt Mackall has convinced them to look at Mercurial, Git is fairly common, Mark mentioned a couple of others.  However, each source control system raises the barrier for participation by the average developer.  His major point seems to be to advocate a standard mechanism for bug tracking, source management, etc., not by having everyone use the exact same tool set, but instead defining some basing API's etc.

In conclusion, Mark encourages greater collaboration across projects (and no, I didn't state this as well as he did, sigh).  But I, for one, agree.

Enterprise Panel at Linux Foundation

2007-06-13T11:14:00.001-07:00

Jim Zemlin asked what the most exciting things are going on now in the Linux space.

Dan Frye took the first answer, focusing on Real Time Linux, working with the Defense industry and various Financial industries, where Linux has taken a clear lead over some of the competition, e.g. Solaris.

Christy Wyatt is focused on the mobility space, with over 3 million mobile users using Linux today.  For Motorola that create excitement and opportunity in the market space.

Chris DiBona is excited about Containers in Linux and resource containment in Linux.

Jim asked how the vendors work with the community and what the opportunities are to work with the community.

Chris talked about how open source usage is very integrated into their workload and how they avoid the redistribution clauses of open source simply because of their typical usage/business model.   Chris also suggested that he could trivially get Speak Up's for Greg KH to do testing based on the accessibility needs brought up in the previous session.

Christy pointed out that Motorola's interactions with the community are broad - 70,000 employees, 1/3rd of which are software developers at an 80 year old company, the culture change is a definite challenge.  However, mobile linux is becoming more defined and is making it easier for Motorola to engage with the communities.

Dan pointed out that IBM uses open source throughout the company, but the LTC's role is to be members of the community - not just "work with the community".  IBM's goal is to remain trusted, valued peers in the community, helping to make Linux better.

Jim asked:  why are you all here?

Of course, three of these are board members who were in for a board meeting the day before, and Chris pointed out that, well, it's here, in my building!  All tongue in cheek, of course.

Dan pointed out that this is where the industry comes together, handling legal, technical, business issues.

Christy pointed out that LF is the center for all things Linux.  She also reiterated that large companies don't want to work *with* the community, but want to learn to be *part of* the community.

I missed the next question, but Chris pointed out that there is more early release hardware available to the community, more interactions early with the community.  There are some people who are combatting the Linux community but the community is also strong enough that those attacks are often ignored.

Christy pointed out that there is no longer concern about whether Linux is a viable choice for the mobile space - with 6 million devices already shipped, that decision is clearly decided.

Dan pointed out that adoption by enterprise of Linux at the "edge" (infrastructure) portions of the business were trivial at most companies - they were less regulated, easier to enter.  Now Linux is being debated for various workloads closer to the center of the workload, multi-million deals in place with enterprise corporations.  There is more to go but Linux has penetrated into the center of the enterprise at this point.

Common Linux complaints today are about ease of use, level of integration, "Fit and Finish" - easy to use.  Uli ??? is a long time AIX/Linux/Mainframe user, how does Linux at IBM compete with IBM?  Dan's answer:  it doesn't.  The markets don't compete, AIX and Linux, for instance are distinct market segments for IBM, the solutions have different strengths.  Dan manages AIX, Linux, VM, VSE, etc. and the channel conflicts are practically non-existent.  Customer needs drive the type of solution that they want and their needs drive their choice of platform, operating system, etc.  Uli points out that internally, people are looking for ways to consider how to move from Unix to Linux and he expected some channel conflict.  Dan says that IBM rarely sees that from the customer - the customer weighs their needs and moves to the appropriate solution.

Uli points out that some of his people want things to work "just like they always did" as opposed to "just work".  Jim points out that IBM is one of the organizations that has long been involved with Linux and the potential for conflict and that there are many IBMers in the audience (/me ducks) who have been dealing Linux and IBM products for a very long time and can provide some insight and guidance for others.

Christy pointed out that Motorola's first product in 2001 was done in China as a way to enter that market.  Christy points out that there is a need to innovate at all layers of the stack to compete, and doing so with 3rd party products is very difficult; open source provides a mechanism for competing, especially in an industry where the software is becoming even more important than the shape and form factor of a hand help PDA/phone.

Eric Wickadella (sp?) from Business Week - is there anything about Linux and Motorola that competes well with the iPhone.  Christy points out that the iPhone capabilities can already been done on many platforms with the possible exception of iTunes (others have multimedia capabilities).  Motorola sort of welcomes the iPhone because it opens up the rich experience and shows where the industry is going.  However, Motorola is likely to drive its strengths based on voice and data plans, software capabilities, etc.  Christy clarified that Motorola doesn't intend to "dominate" the industry/stack - obviously like any company they will compete on strengths in that space.  Eric also asked about Safari - Christy thought it was useful, but it doesn't really cover 3D performance/gaming.  Messaging being the #1 capabiity and gaming be #2.

James Bottomley wants to collaborate:  what is the top 2 list of what the community could do for each of the vendors and the top 2 list of things where the vendor can contribute back to the community.

Chris wants to contribute back to the Pacific rim (Google helping the community, the community helping Google).  Also, he'd love to see the community create drivers for things like nVidia & ATI drivers.  The proprietary drivers are one of the significant thorns in their side.  [See the nouveau project, the vesa and, um, mesa? projects for work in this space --gerrit].

Christy points out that most mobile users don't realize that their phone is running Linux.  Consolidation and collaboration across the ecosystem to simplify and make more common the user experience on cell phones.  Motorola can offer back training of the mobile operaters about how Linux and open source works and is acceptable for use in that space.

Dan doesn't usually rely on the community to do things for us, but prefers to contribute directly and work with the community when we need something.  Request #1 would be:  when GPLv3 comes out, just "chill"!  A bit of laughter on that one.  ;)  Power management - that is where the community needs to do better.  Patience and persistence on the device driver front, helping to train and work with the stubborn, proprietary device owners.  Christy seconded the power management issue.  Chris wants to second the GPLv3 "chill" request.  Chris quoted Eban, roughly transcribed as:  "GPLv2 wasn't perfect until we started GPLv3 development."

Jim:  what do the panelists expect out of this meeting?  Dan:  continued collaboration, hopefully some specific work actions out of the meetings with a focus on making Linux better and building the industry.  This is important to the industry and we need to keep at it.

Christy:  reemphasized that the US/Europe locality comment from earlier in the day was important - most of their developers are actually in Asia.

Chris:  we take great benefit from Linux - when you leave here, have something in your mind to *do*, be it publishing, using, developing.

Linux Foundation Collaboration Summit

2007-06-13T10:31:00.001-07:00

An exceptionally full house at the Linux Collaboration Summit with nearly all the names that matter.  Current panel has Andrew Morton, James Bottomley, Chris Wright, Ted T'so and Greg KH, moderated by Jonathan Corbet.  See the web site for the full agenda for details.  Jonathan's first question was related to the stability of Linux, which led to a fairly lively discussion discussion.  Even Christoph Helwig had some heckling, er, good points to make, including the fact that the rate of change has led to less community review, and the fact that there is still no regression test suite.

There are several forces pulling in different directions, leading to a lot of tension on the core kernel, including support for new hardware, stability for old hardware, support for new features, stability for commercial end users, continued subsystem clean up, purity and simplicity of design, ease of mainenance, etc.  Also, the "scratch your own itch" philosophy of code development leads to a lot of new and conflicting technologies moving in parallel to mainline.

Mark Shuttleworth started and engaged in a discussion where Andrew Morton requested more detailed input from the distros, including problem areas, proposed release dates, staying closer to mainline (perhaps), and better dealing with API stability (and how that causes divergence from mainline and related stability).  There was some applauding of the fact that distros were pushing back on accepting changes that were not also and already upstream.  That helps maintain a single source for bug fixing, stabilization, and a location for hardware enablement.

Don Marti from LinuxWorld.com asked about Andrew's State of the Union comments about Linux's lack of Power Management and for an update on the state of Power Management and what help others could provide.  Today most power management in linux is binary - on or off, where many vendors are now providing low power or reduced power states.  Andrew strongly requested access to specifications to devices in general, specifically here for power management APIs but generally the community has requested APIs for devices in general to help make the devices operate optimally on Linux.

Jonathan pointed out that a recent driver he had written did not have suspend/resume capabilities and the community did not point out the need for power management as a basis for accepting a fully functioning driver (good point) but Greg KH pointed out that the community will accept drivers at any stage of development, assuming that they will evolve over time to have more complete functionality.  Jesse Barnes from Intel pointed out that there is not a robust, complete API or environment for people to program to.  Also, Jesse reiterated that specs for devices are good - but he would also like to see vendors write their own Linux drivers for their hardware.  Greg KH pointed out that he has 85 driver writers standing by, ready to write drivers for devices that vendors contribute and provide specifications to.

David Slessinger from Axxess pointed out that mobile/open source activites are more prolific now but Chris Wright countered that the mobile folks are not very visible to the kernel community.  More engagement, visibility and interaction there would be helpful.  Greg KH pointed out that servers have the same power management needs as the mobile community and the two communities need to work together better (server and mobility) with respect to power management.

Ted suggested that an interface to broadccast that a disk was just spun up could be used to communicate to other subsystems that the disk is available, power has been expended, why not opportunistically do any other current activities that need access to a spun up disk?  Andrew pointed out that he has not seen requests, requirements from the mobile community and really doesn't know what capabilities that they really need.

Accessibility workgroup chair, Janina Sajka is present and mentioned that a screen reader patch is available for the kernel but Greg KH pointed out that the patch needs a lot of work and it needs some testing - however he doesn't have access to the hardware to test.  Janina promised to work with him to get a coder and tester to get the code into the kernel.

Darren Dagless (sp)? from Novell brought up ATI & nVidia proprietary drivers.  Novell has a lot of proprietary driver partners and Novell does not support those drivers directly.  Christoph suggested to tell them to "go away" - however we do want access to the hardware that some of those vendors provide.  This area is as contentious as always - with the primary viewpoint of the kernel community is that vendors need to provide open source drivers or users need to avoid the use of those hardware.  Ted also pointed out that both individual users and system vendors integrating hardware into their systems need to be conscious of the open source driver availability of the hardware components that they use.

Greg KH pointed out that the Linux Foundation has a mechanism for allowing vendors to provide specifications to the Linux Foundation under NDA and still get a driver written and supported by the community.

Stephen Von Nicholson from Ziff-Davis talked about how ATI has now offered up specifications even if not drivers.  On June 29th 2007 the GPLv3 will go live.  He asked what the kernel community now thinks about GPLv3 (earlier some had claimed that the GPLv3 was the end of free software as we know it).  Greg KH still believes that the GPLv3 has not addressed the primary problems and is not a fan of it for use with the Linux kernel.  James pointed out the the logistics for converting to GPLv3 are probably also prohibitive because there are no localized ownership for copyright, etc., and the benefits of GPLv3 are not likely to be compelling for a change that would be logistically overwhelming at best.  Ted pointed out that the GPLv3 has improved dramatically over the past year or so, and he is not as opposed to the use of GPLv3 for user level applications as appopriate.  James and Ted point out that Eban Moglen was very responsive to the problems pointed out by the kernel community (among others).  And, there is now one more license in the ecosystem and they will all over time co-exist.

Someone asked if the kernel could be converted to GPLv2/GPLv3 dual licensed.  General answer was "not easily" and "not likely".  Don Marti pointed out that 1/3rd of the kernel is already licensed (in source code comments) as GPLv2 or later, as opposed to GPLv2 only.  Also, much of the kernel is dual licensed today, especially in the driver space (e.g. drivers used in both Linux and BSD is pretty common, with different licenses).

Jonathan posed a question about contributors - lots of US and Europe centric contributors, lots of desktop and server vendors, but not much from Asia, not much from mobile and embedded companies.  His question focused on how to expand the contributor community to include a more representative, diverse set of companies and interests.  Andrew pointed out that some vendors have not seen a need to interact with the communiy, in part because of the high rate of change of their products, and the need for a longer term involvement with the kernel community.  James pointed out also that the discussions are all in English, so non-English speakers are at an inherent disadvantage.  There is an LF collaboration summit in Japan already, James suggested one in India or Africa would be possible.  Also, the existance of a "universal translator" might help, with a couple of chuckles aimed at the Google folks present since they seem to be active in spaces like this.  Chris pointed out that the developer community has not been getting as much "new" membership lately, especially in the core contributor space.  Ted pointed out that the OSDL/LF sponsored Japanese collaboration summits provided simultaneous translation to/from Japanese/English which substantially helped collaboration at those summits.

That ends the first session, and I'll do more if my battery, fingers, and attention span all hold out.

Ongoing study of Free and Open Source Software Market Share

2007-04-13T18:51:00.001-07:00

I don't have much to say on this report since it is long and I haven't it all yet but it looks to be pretty interesting for those wanting to know how Open Source is faring in general in the market place.

Virtualization ready for serious tire kicking...

2007-03-20T14:34:00.000-07:00

With Red Hat's announcement, Xen based virtualization is now available pretty much anywhere you'd want to use it. In addition to many/most variants of Linux, Solaris also is enabled for use in Xen's domain 0. There is still some work to do in mainline with paravirt_ops to make any linux kernel easily or trivially configurable for use as a domU (guest) operating system but most of the work for paravirt_ops initial uses is complete.

For those looking to consolidate workloads, test new environments, rapidly prototype new workloads, and perhaps even experiment with application containers, this is a good time to do so. From here on out, virtualization should just get easier and easier to use on Linux.

Venture Capitalist funding on the rise again

2007-03-20T14:32:00.000-07:00

According to this article, venture capital funding is again on the rise. Health Care is definitely a focal point for funding (much needed, IMHO). Of course, one of the challenges will be deploying solutions which are not just expensive "one-off" solutions, where every deployment in any geographical or logical area is unique to that area, and thus more expensive. Finding open source solutions which are broadly applicable across the health care industry, as well as help drive more accuracy in diagnostics, more follow up in referrals, and a better holistic picture of our individual (or family) health would be an excellent area to help drive down costs in health care while raising the overall quality of health care.

Ian Murduch goes to Sun

2007-03-20T09:40:00.000-07:00

Ian Murdoch is (and has been) the lead for the Linux Standard Base (LSB) effort which attempts to standardize Linux APIs and ABIs for applications. Also, as a result of the merger between the Open Source Development Labs (OSDL) and the Free Standards Group (FSG), Ian was briefly the Chief Technology Officer for the resulting Linux Foundation.

Ian was snatched up by Sun this week. There are numerous rumors about what Sun might get out of this, including LSB branding for OpenSolaris on x86-64, addition of Java to the LSB, or certification of Sun's Java as an LSB certified application. Sun has a mixed history of working with Open Source, with positives being OpenSolaris and more recently choosing to license Java under GPLv2.

Ian's move obviously creates an opening at the Linux Foundation for a CTO. That position will likely be filled quickly to aid with the continuing/ongoing transition to the newer, more streamlined model that LF will be using for protecting, promoting, and standardizing Linux.

Software Appliances on your iPod?

2007-03-20T08:57:00.000-07:00

There's been a lot of hype about software appliances - basically pre-packaged applications which provide a fully encapsulated, targetted solution. Some of them have the advantages of reducing testing by packaging a tested operating system, libraries, necessary middleware (e.g. the version of Java that *works* with the application), and all the appropriate configuration files so that the applications "just works". Another use of software appliances is the use of a full system that you can carry with you, plug into any computer, "boot" your image, and use all of your normal applications, data files, etc. This solution from moka5 seems to be in the latter camp, where you can put your entire personal image on a USB key, or, more fun, on your iPod, carry it anywhere, boot from it on your friend's PC, etc.

I believe there will be some move that allows your personal PDA, phone, iPod, whatever to be your home image, allowing you the ability to use a larger display, keyboard, etc. whereever you happen to be, but without requiring you to carry the bulk of a laptop. And, with configurations like VPN's, personal apps, email, you can carry your world in your shirt pocket or purse.

OSDL and FSG merge to create the Linux Foundation

2007-01-30T23:28:00.001-08:00

Okay, I really like the name "The Linux Foundation". It reminds me of Knight Rider's Foundation for Law and Government, or maybe Isaac Asimov's Foundation series. It just sounds cool. And Important. Well, I don't know if it is really *that* important overall, but it will serve a necessary function in the overall Linux ecosystem. The Foundation's goal will be to promote, protect, and standardize Linux. Promotion mostly means marketing and awareness of Linux's existing capabilities, although I believe over time it will also include identifying gaps in capabilities which are inhibiting Linux adoption. Protection is an obvious and easy reference to the education of the legal impacts of Linux and Open Source, but also has a strong focus on Licensing and patents/intellectual property protection. And standardization refers to the existing LSB efforts which will likely continue to grow (slowly) over time. LSB standardization is a slow process and technically challenging, as well as limited by adoption by distros based on their willingness to claim the latest and greatest level of support.

Overall, in some ways it looks like the little fish is trying to swallow the big fish. FSG had a pretty small budget and a pretty limited scope. OSDL had a huge budget and a huge scope. But FSG tackled a small, hard, well defined set of problems. OSDL tried to tackle a very large, nearly unbounded set of problems. The merging of the two extremes should be good for both groups. Both are going through a little bit of culture shock, I think but all seem to be working hard to find a good working balance. I think overall this will work out for the best for Linux and Open Source in general.

The most interesting thing was a recent meeting in Palo Alto which brought together a few community members from the Linux kernel community, a bunch of vendors, and several large, corporate end users. The culture clash was apparent, even though all were on their best behaviors. But the goal was to bridge those culture gaps, make the groups aware of each others pain points, and ultimately lead to a positive working arrangement between the extremes. We'll see how that works out over time, but the first painful little step has been taken. I'm curious to see how this meeting of the extreme poles of Linux community and end users works out over time. I have high hopes, though.

Mozilla, IceApe, Firefox, Iceweasel and Open Source re-branding

2007-01-30T23:01:00.001-08:00

Recently I was a victim of one of the "strengths" of open source. Specifically, if you don't like the direction a particular project is going, CLONE it. Go your own direction. Become your own Maintainer. Pray that you have the user's to validate all that work you are about to do.

I'm a, um, loyal?, Debian user on my laptop. That loyalty is perhaps, a bit more laziness than true loyalty. I often believe that I should be transitioning at least to Ubuntu but really haven't had the urge to figure out how to gracefully get there. I know Debian as a project, I understand apt, I normally sigh and smile at the sociopolitical views of Debian and sometimes even believe that aligning with those views is to be one with the great purist views of open source philosophy. How Zen (not Xen, this time ;-)).

However, I spent a painful week or so as I learned about Iceweasel and Iceape. What are those, you might ask? Well, nothing more than rebranded Firefox and Mozilla, respectively. Now, I don't claim to really know about or care about the licensing issue that tripped things up here. But, despite the pain of a non-working Iceape and Iceweasel for nearly a week (I'm still not sure I understand why, although I think update-iceape-chrome was the magic trick at the end) I got to see open source in action.

Specifically, when a project no longer aligns with your particular goals, you can clone the source. You can modify the source, you can develop in a new direction. You can start from a position which leverages all the existing efforts in community development, modify to fit your needs, use it and/or make the results more widely available (including making source available!). You can do it without consulting anyone, you can do so without fear of infringement. The good licenses allow you to copy, modify, release, without the cost of starting from scratch.

Over time, this capability of open source will simply grow the ability to create or enhance applications at very little incremental cost to each person that adds value.

Now there is some real heavyweight development power.

Open Source Driver Development

2007-01-30T22:43:00.001-08:00

So, Greg KH and the Linux community are offering free driver development. This is very clever, I think, and potentially a very cool thing. The is interesting on many levels. On one level, the availability of free drivers enables hardware vendors/IHV's to reduce their costs. On another level, it means that the must admit that their hardware sales are their revenue input and the software is merely an enabler for that hardware revenue. In other words, software has no real economic value, and usually has a large development cost that can be avoided. On a completely different level, it means that the IHV's must release specifications to their hardware to the public (yes, there are some ways to restrict that, although odds are they are a hassle which is really not worth the hassle) and the level of competition in devices shifts from "secret ways to make hardware work well" to "public specs which have more features with great hardware support". In other words, I think public specs will drive more direct competition in real hardware value. Of course that means that vendors will be forced to have real value in their hardware which could be a good thing for the consumer.

All in all, the real proof will be in the pudding, as they say. Until a few hardware vendors show that this is a viable option, there will likely be some skepticism. Going open kimono on hardware specs scares a lot of IHV's - it means that they show their (often small!) hardware value add to their competitors on their public specs now. And, there is the risk that the rather judgemental nature of the Linux community may still avoid the use of more advanced capabilities. For instance, even releasing specs on a network card that supports TCP Offload Engine (TOE) support may not get full community support.

But, really, I think this will be a great thing for the Linux ecosystem overall. Provided someone signs up....

Linux Desktops getting easier? Or Microsoft Desktops getting harder?

2007-01-02T17:28:00.000-08:00

This was an interesting article given some of my past comments on the Linux Desktop. I'm beginning to wonder if it is a race for Linux to get better, or if it is just a matter of waiting until Microsoft hoists itself by its own petard with its convoluted and ever more restrictive licensing. I don't think Linux won here because it is perfect on the desktop, but it clearly reached that cross over point where it was better at *something* important.

To be honest, Linux has had some similar problems (Adaptec and the infamous SAS driver comes to mind) but now when people hit problems like this, they have to stop and think: Is it worth it? What are my options?

And Linux is now truly an option to consider.

MS and Novell

2006-11-03T13:20:00.000-08:00

Well, after Oracle's recent announcement about providing support for Red Hat customers, the Novell/Microsoft announcement seems to add more excitement to an already hyperactive open source environment! Red Hat has a large rebuttal to both that is pretty informative. There are plenty of analysts out there who think everything from "Microsoft Embraces Linux - Extends capabilities" (on the way to Extinguishing Linux in the traditional Microsoft Embrace, Extend, Extinguish model?) to debates about the values to Novell, the challenges to Red Hat, the potential alliances being formed, the legal lines being drawn, the benefits to Microsoft, etc.

Clearly, Microsoft wins some points from this arrangement. They win money from Novell (royalties for something or another), they prevent Novell from asserting patent rights (rumor is that Novell has a few that might read on portions of Microsoft's .NET or other application technologies), they establish themselves as an Enterprise capable, viable virtualization platform supporting Windows AND Linux (except they haven't shipped their virtualization product just yet), and introduce some standard FUD into the Linux market place.

Novell in the short term wins. Microsoft obviously can ship a lot of additional vouchers for SUSE Linux, leading to more revenue for Novell (those were discount vouchers, right? So there is still *some* software revenue), Novell doesn't have to worry about legal action from the land of Microsoft for a while and can presumably focus on technology and sales. Novell gets a lot of other press and is recognized as a potentially legitimate peer of Red Hat - something it has been struggling with since the acquisition of SUSE Labs. And it adds another Enterprise capability to its sales literature "Works with Microsoft" which is clearly appealing to many Enterprise class customers.

In the short term, the impact on Red Hat is harder to guage. I think this is a case of "what does not kill them, makes them stronger.". As long as it doesn't kill them! I personally think the technologists and agressive marketing team there will survive this, will polarize the Linux market place a bit, and encourage a bit more competition in the US at least, which can only make both Novell and Red Hat stronger over time. And, this (in combination with the Oracle announcement) might make Red Hat a bit more responsive to their customer base, which I'm sure will wind up pleasing their customers more over the next year or two.

And, for Linux in general, it is a clear sign that Linux and the Open Source software stack above it is a market contender that now gets to play with the big boys. And, in that game, I think Linux will continue to grow up and emerge as a winner in the long term.

Healthcare Illusions

2006-09-30T15:09:00.000-07:00

I just read this on my Starbucks coffee cup:

"One of America's most cherished political illusions is that we all receive the same healthcare regardless of income. Another is that we don't ration healthcare. The reality is very different. A change is needed and we have the power to bring it about."

Dr. John Kitzhaber (also on Wikipedia)

Former governor of Oregon and healthcare reform advocate

Dr. Kitzhaber is the founder of a new movement known as the Archimedes Movement which states in its opening comments a quote from Archimedes:

"Give me a lever and a place to stand, and I can move the Earth."

The Archimedes Movement currently has a draft of a legislative proposal to help address the current crisis in health care. It is currently open for comments.

Free 376-page book on migrating to Linux

2006-09-20T10:58:00.000-07:00

What timing - I just ranted a bit about the dichotomies in the Linux Desktop last night and this morning I get a link to an IBM Red Book which was authored by one of my friends driving our desktop strategy. The overview here looks pretty interesting - I haven't read the book yet but hope to soon.

Here's the announcement that went out with the book:

A final draft of the second edition of IBM's "Linux Client Migration Cookbook" is now available online for free downloading. It targets "IT environments that need to begin an evaluation of desktop Linux, or in a broader sense any organization whose strategy is to move toward greater adoption of open source software and open standards."

read more | digg story

Linux Desktop Dichotomies?

2006-09-19T14:38:00.000-07:00

Several years ago I was somewhat amused by the core Linux communities extreme focus on Linux on the desktop. More specifically, the focus was on Linux on *their* desktops because, after all, Linux was by the geeks, for the geeks. At that time, I was interested in enabling linux to run on larger hardware, as were several other folks working for vendors of larger hardware platforms. The initial and, even somewhat today, continuing resistance to scalability enhancements seemed painful at times - the ongoing refrain was that scalability could not hurt performance on desktop platforms. In other words, the primary focus of at least the kernel community was on the desktop systems.

Several years later, there are a number of blogs, papers, analysis like this one, which continue to show that Linux adoption on the server side has far surpassed the adoption rate on the desktop. There is some irony in the fact that the primary focus was on desktop for years before there was any focus on servers, and that in some ways the focus on desktops possibly slowed down the rate of development enabling high performance servers.

A further irony is that the standards for scalable code remained in general so high that I believe desktop performance improved as well from many of the enhancements designed for larger systems. Add to that that most of the server software takes advantage of a few key kernel APIs in the networking space, the locking space (futexes, epoll, etc.) that server and infrastructure applications tend to perform quite nicely on Linux. At the same time, though, most desktop software has become larger, more complex, more divergent on Linux, resulting in lower adoption rates of the Linux desktop and related client software.

This seems to be a case where the desktop communities have had a different and distinct focus from the Linux kernel community, who, in many cases actually enabled a richer set of desktop client software because of its general availability, low/free cost, full development environment (for free), etc. I don't completely understand how the desktop community failed to achieve as much market adoption overall as the server community did. There are probably lots of reasons, too numerous to go into here.

The good news is that there is a lot of focus now on Linux on the Desktop. Everything from the extremely enjoyable "Why Userspace Sucks" by Dave Jones (LWN coverage is great!). His slides (tar.gz) are available as well. Then there is the Portland Project which has a wiki where they are trying to find commonality in some of the basic services offered on the desktop. OSDL has also Desktop Linux initiative which is looking at the big picture, basically understanding all of the related open source communities, identifying gaps, finding commonalities, addressing ISV porting issues, etc. And, as part of the Desktop Linux initiative, they have set up a group of desktop architects as the focal points for these discussions.

Now, all of this work in progress doesn't mean that the solution is at hand or that it is easy. However, judging from reports on Novell's SLED, Red Hat's Fedora Core 6, Ubuntu & siblings, etc., there are a number of usable desktop environments today. One of the key problems that these desktops have been fighting is the old battle cry "But it isn't Microsoft!" Well, yeah, right, it isn't. Maybe in many ways the Linux desktops are better, in fact. Well, and there's the rub - there isn't just *one* Linux desktop that one can compare to Microsoft. There are many. And they are all different. And they don't have as many packaged applications on the shelves at Walmart or Costco. And, for various reasons, the Linux Desktop hasn't reached that "tipping point" where it gains a respectable following of ISV applications that are on par with all those specialized apps on the shelves at those high volume retail or wholesale stores.

The past three or four years have been "The Year of the Linux Desktop!" Are we closer? Not according to the sales/marketing/installation numbers. Are the numbers wrong? Maybe. But are they *that* wrong? I don't think so. Linux desktop adoption is definitely dragging, despite the rapidly maturing capabilities of the Linux Desktop.

Maybe 2007 will be the year of the Linux Desktop...

Software Patents and Open Source

2006-09-19T14:06:00.000-07:00

It is amazing what daily meetings can do for productivity. There is a direct correlation between my blogging time and my additional meetings, I fear. But, I've been saving a backlog of interesting tidbits worth some commentary.

For instance, the latest one involves one of the open source heroes - well, he's a champion of sorts, if you consider the most radical proponent for overhauling our beliefs about intellectual property to be a hero. Many do. But RMS is usually a bit more extreme and radical than our global communities are able to handle. Take for instance this article, and his stance on OSDL's work on improving the (software) patent system. Part of RMS's assertion is that software ideas should not be protected at all - they should be freely shared. I'm not convinced that assertion is ideal for promoting small companies and new developments in a larger macro-economic view, although I also view the patent protection system as being unable to keep up with the times as it stands.

OSDL's effort is currently centered around improving the screening for patents that get approved. A logical step for improvement when an in depth analysis of many existing software patents can easily create debates about whether those patents had been previously implemented before and were thus not novel (the two basic premises for a good patent is that it is novel and implementable). Today, the US Patent Office only looks at a small set of sources, one being their own issued patents, for determining if something is novel. OSDL is asserting that Open Source has allowed many people to innovate without filing patents, and that source of innovation should be considered when examining the novelty of a patent. All in all, a reasonable goal, I believe.

I also believe the more fundamental question of whether the USPTO (or other patent systems world wide) should allow patents on software is still debateable. But the real value there has to be compared to the USPTO's original mission as defined in the Constitution, which (from the USPTO site) says:

The Constitution of the United States gives Congress the power to enact laws relating to patents, in Article I, section 8, which reads “Congress shall have power . . . to promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries.”

So, do software patents promote the progress of science and the useful arts? Or do they stifle innovation?

I think the jury is still out on that one.

Containers vs. Xen: Everyone wins!

2006-08-22T18:36:00.000-07:00

So containers are starting to get some press now. This seems to be a pretty fair issue. As I mentioned earlier in my blog, there is actually a mailing list where OpenVZ, vserver, and a number of community folks are actively engaged on bringing Containers into mainline Linux. Yes, the performance benefits of lightweight containers are great, the ability to do some resource tracking and throttling of containers will be useful for many end users and No, containers will not replace Xen. Xen provides a strong isolation model at increased overhead. Containers provide a low isolation model at decreased overhead.

Of course, with Containers, you have several options today to choose from, each of which have their drawbacks. vserver provides pretty complete isolation, has a great community supporting it, is available in some of the community supported distros, etc. There is also OpenVZ, supported by the SWsoft guys directly. And there are things like OpenMetaCluster from IBM, which can checkpoint/restart databases on clustered systems, although with a little more limited availability. All of those folks are working to get a common solution into mainline Linux, which should happen in time for the next big cycle of enterprise distros. Of course, in the meantime, the distros have each chosen a champion that they are considering. Since the changes for either solution are fairly invasive, they are outside the normally accepted changes for a service pack or update release. But in the meantime, let the fun continue!

gerrit

Open Source Goes vertical

2006-08-21T13:45:00.000-07:00

Bernard Golden was one of the panel moderators at the LWE Health Care Day. He did a very nice write up of his panel on his blog. In his blog Bernard looks more at the flow of VC capital and start up companies. I think a more reasonable short term model will be to get some support from partnerships of existing large open source players (e.g. the system vendors) and the large health care integrators, hopefully working directly with customers and maybe some new VC companies to help fill out some niches. The real goal here is to build a community which can help address the problems of IT availability in Health Care, not just a few new, smaller companies.

gerrit

New Health Care open source mailing list

2006-08-18T12:23:00.000-07:00

The wide spread interest from OSDL's Health Care Day in building a "community of communities" around Open Source in Health Care has motivated OSDL to set up a new mailing list for all interested parties. If you are interested in Health Care in the Open Source space and cross pollenation between various communities, you may want to join the health care community mailing list.

Gerrit

Overview of the state of Health Care and IT

2006-08-17T16:50:00.000-07:00

As part of organizing the OSDL sponsored Health Care I had the privilege of meeting and talking to some very bright folks in the Health Care industry. One of those that summed up the status of IT in Health Care and the potential impacts of poor IT on human life was Dr. Kenneth Kizer, CEO of Medsphere. He previously was involved in the transformation of the Veteran's Administration health care system into the largest and one of the very best health care systems in the United States. His OSDL Health Care Day presentation is very insightful and well worth a read.

gerrit

Eclipse OHF, Health Care and Open Source

2006-08-14T15:42:00.000-07:00

Just in time for Linux World Expo IBM came out with a press release about IBM's involvment in the Open Source - Eclipse open Healthcare Platform (OHF). Thanks to Eishay Smith for the info!

Containers and Virtualization

2006-08-14T15:36:00.000-07:00

The sourceforge list for containers (lxc) is being shut down since sourceforge mailing lists are being soooo slow. The list list is at OSDL - containers@lists.osdl.org. People working on openvz and vserver (with the goal of getting a common solution into the mainline linux kernel) will get postings sent to this list as well since the plan is currently to subscribe the existing openvz and vserver mailing lists to this list

To subscribe, please visit:

https://lists.osdl.org/mailman/listinfo/containers

The end result of this discussion and relevent patches should be a single infrastructure in the mainline kernel that is fast, reasonably complete, and allows any other solutions to be wholly contained within the kernel or enabled through user level applications on top of the base kernel support. Today, all major groups are consolidating here with some healthy debates about the details of implementation. No outlook yet on completion but all groups are motivated to generated patches as fast as possible.

gerrit

Open Source and Health Care

2006-08-14T12:06:00.000-07:00

Another activity I've been involved in lately is looking at Open Source utilization in the Health Care field. I have a lot of reasons why I think this is very interesting, which I'll try to share sometime in the future. However, for those that may be interested, I'm in San Francisco this week at Linux World Expo and participating on a panel at their Health Care Day. I should have lots more information to share about where Open Source and Health Care are headed after that event as well. Feel free to look me up if you are in San Francisco!

Virtualization

2006-08-14T11:35:00.000-07:00

Here's one of the more accurate articles I've seen on the current state of Linux and Hypervisors.

I spent some quality time with Simon Crosby (XenSource) and previously with Jack Lo (VMware) as everyone was working to find a good solution to getting Linux to directly run on a virtualized platform. Of all the news articles on this that I've seen so far, this is definitely the most accurate.

On a similar note, whether or not Xen is ready for prime time, we at IBM have done a lot of testing in support of Novell's inclusion of Xen and believe that it is ready for Enterprises to begin testing and evaluating for use in Enterprises. It is still relatively young in capabilities compared to System p and System z virtualization capabilities, but it seems to provide the basic capabilities for IA32/x86-64/EM64T systems quite nicely. This article calls out Novell's position at this point.