Ksplice: reducing the need for kernel reboots

Ah, good morning...  This morning I'm moderating a session on New Technologies - the first presenter is Jeff Arnold who is covering Ksplice.  From the LF Program:

Ksplice

Jeff Arnold, Lead Developer, Ksplice
Today, when Linux users apply security updates to their systems, they are commonly informed that they must reboot in order to finish applying the updates. Since rebooting is disruptive, many users and system administrators delay performing these updates, despite the increased security risk--more than 90% of attacks exploit known vulnerabilities.  New technology out of MIT, called Ksplice, enables running Linux systems to stay secure without the disruption of rebooting. This talk will describe the core innovation behind Ksplice and how this technology will improve the security and maintainability of Linux systems.

As good as Linux is (transcendent?  ;-) it still has bugs.  Some of these include security problems which typically need updates as quickly as possible.  However, people do not enjoy the scheduled downtimes, especially in this world of round-the-clock access to services.  Reboots also have the problem of losing software state such as network state, application state, etc.  90% of attacks exploit known vulnerabilities, so getting patches installed more quickly would reduce the level of vulnerability.  Also, applying patches sooner avoids the need to wait for a scheduled maintenance window.  Ksplice currently adds negligible performance impact.  If any impact is found, please notify Jeff asap!

Ksplice works on any release since Linux 2.6.8.  he initial release is in April 2008 (GPLv2).  Ksplice has tools today in Debian sid, Ubuntu Jaundy, Fedora 8-10.  It is currently proposed for mainline and has 5 engineers working on Ksplice full time.

Question from the audience - doesn't virtualization obviate the need for this type of technology?  Answer:  Not really.  Even the unlderlying OS supporting, say, KVM, still needs to be patched.  The impact of downtime or mandatory migration for all virtual machines running on that physical machine is still extensive.  And, taking down the network state or application state for a virtual machine still impacts the workload.

Jeff provides an example of code that needs to be updated, and shows a way to apply a patch and generate a ksplice patch.  He then shows a binary exploit, followed by a complex patch including asm code and C code to a couple of files.  Then uses Ksplice to build two kernels, one with and one without patch.  Then with a simple sudo ksplice-apply applies the kernel patch, ksplice-view shows what ksplice updates have been applied, and a re-run of the exploit code fails to provide him with root privileges. ksplice-undo allows the patch to be removed.

These changes only update the running kernel.  However there are also tools which update the kernel for the next reboot or allow a set of rc scripts to apply updates at next boot to the same kernel.

Source for the kernel must be available to apply the patch.  If you try to build and install against the wrong source version Ksplice will notice if the kernel changes.  You must also use the same compiler and assembler version.  Could a vendor provide enough information so that an end-user could create these patches without the full source? Possibly yes.  Or could a vendor build these kernel modules for distribution to end users to allow the end users who don't have full kernel source easily available?  Yes.

Jeff showed a Ksplice Uptrack Manager much like the Ubuntu software update menu.  Looks like like a normal package manager but with patches getting spliced into the kernel?  The kernel stops for about 700 microseconds (0.7 milliseconds) during this update.

Question from the audience on what the likely model is here for distributions.  Does every vendor need an infrastructure for their patches.  Ksplice is hoping to be able to create these updates for a number of distributions.  A distro vendor member would like to have both the mechanism for updating any set of pre-build kernels as well as the running kernel.

Next Jeff talks about the mechanism for changing the code based on conversion from a source code patch to the binary updates.

Identify which functions are modified by the source code patch and then generate a "replacement function" for ever to-be-replaced function and then start redirecting execution to the replacement functions.

pre-post differencing.  Pre source is unmodified, post-source is the post-patch application source code.

Ksplice compiles both pre and post, then compares the two binary object files and extracts the replacement functions.

Shows a picture of foo() and foo()' (foo() prime), adds a jmp instruction at the beginning of foo() to the beginning of foo()', return from foo()' returns to caller of foo().

There are some symbolic references that also need to be patched as well. 

Need to make sure that foo() and foo()' are not running at the same time.  Temporarily grabs all CPUs, makes sure it is not running one of those functions, if necessary, abort(rare).

Design only allows changes to code, not data structures today.    ksplice_pre_apply() and ksplice_post_apply().

In some cases, a patch may need to iterate over all CPUs or use the ksplice_apply() call directly to apply the patch.

Hypothesis:  most linux security patches can be hot-applied with Ksplice.  Used all security patches from 2005-2008, and they were able to hot-apply with 88% without change, and 100% with only about 17 lines of code per patch of additional code.

Debugging with Ksplice can do some of what kgdb and SystemTap can do because it does live patching.  Can insert code almost anywhere and discover any symbol value.  x86_32, x86_64 and ARM.

Jonathan asks how we avoid making the problem worse by creating new versions of a kernel that was never really seen by a distro or a kernel developer?  Is the concern that there is unreviewed code?  Or is it that there are a larger variety of running kernels modified "in place".  You could use Ksplice for just the 88% of directly applied changes without having some of the extra code (which could itself have bugs) and limit some of the risk.  Any new code for the remaining 12% is intended to be as absolutely small as possible to minimize the chance of introducing new bugs.

Is this a portable root kit?  Well, sort of if you have root already and can load modules, yes.  Does it make it a little easier to generate a module which can be loaded to root the machine?  Probably.  But black hats already have these techniques and it gives the white hats an easier way to keep up with the black hats.  It is possible to also use cryptographically signed modules with Ksplice as well to improve security overall.  There are also a lot of tools to see what ksplice updates are installed, sets taint flags, etc. to make sure that people can tell what was changed.

This should work well with the stable trees as well although they've had to remove some patches to make a stable tree do a live update to a running base kernel.