domingo, 16 de octubre de 2011

9p-srv: experimental devdraw for p9p

From the beginning, one of the goals of wsys was to replace all the system dependent code duplicated in several Plan 9 related projects. The unix version of wsys could already do this from Plan 9 (9vx or drawterm) and Inferno but, until now, there was no way to use wsys from p9p programs.

The problem with p9p is that it does not use common 9p devices. Instead, it uses a modified libdraw and the devdraw(1) program. There are versions of devdraw for x11 (x11-*.c) and several OS X systems (osx-*.c, cocoa-*.c). The file 9p-srv.c (included in the util/ directory of the wsys distribution) is an implementation of devdraw which makes use of 9pclient and wsys.

Compile 9p-srv.c and set the DEVDRAW env variable accordingly and wsys will take care of the windows of p9p applications too.

NOTE: 9p-srv is an experiment, even more than wsys is. It can fail and, anyway, it probably is not the best solution to the problem. It was an easy way to get the ball rolling, and it gives me more chances to test wsys.

devwsys is now wsys

Rio is a file server which provides virtual devices, but it is not called a device. Other file servers offer some other files which can be mounted on /dev, but they are not called devices either. P9p's devdraw is the exception, not the norm and, therefore, devwsys has been renamed to wsys.

For the moment, the code has not moved from its usual location: https://bitbucket.org/yiyus/devwsys-prev

There has been several improvements since the end of GSoC. Check the changelog.

domingo, 21 de agosto de 2011

Manual pages

man(1) pages have been added to the devwsys-prev repository. Html versions are available in the wiki of the project at bitbucket. The new manual pages are:

Wctl: a killer service

In Plan 9, rio(1) does not only manage windows. In fact, it is also a process manager. We create a new window to make a new process. When we want to finish it, we delete that window. Devwsys, on the other hand, is running on a host system, and therefore it cannot create or delete processes.

What we need to create processes is a service in the hosted Plan 9 system which mounts devwsys and runs the given command. This is what the wctl script does, announcing a file descriptor in /srv and reading new commands, in the format of rio's wctl service.

A more interesting problem is how to finish a process when that window is deleted. Rio reads a pid as part of the attach specifier. When the user asks to delete a window (with the mouse or a wctl message) a different thread sends a note to the process, waits a bit, kill it and, only then, deletes the window.

Devwsys cannot create or finish processes, so it needs a method to say it to wctl. The IPC mechanism used is, of course, 9P. The pid in the attach specifier is read by devwsys and stored. When the user wants to close a window (with a wctl message or a request from the X window manager), a read of the file /kill will return that pid. If we read the file before the window has been asked for deletion it will just block. That way, we can wait for a window to be deleted. If there is nobody blocked reading /kill, the window is immediately deleted.

The "killer" process has access to the kill file mounting the root of the devwsys file system, using the attach specifier "/". If this specifier is given to mount, a new window will not be created. Instead, we will have access to a file tree containing the kill file and the wsys and draw directories (where the draw directory does not contain a new file, because there is no associated window).

The first thing the wctl script does is mounting the root fs. Each time a pid is read from the kill file, killproc will send a kill note to the process, wait 0.2 seconds, and then will send kill to the ctl file of the process. Once the process finishes all the references to its window file system are lost, and the window is destroyed by devwsys (in theory, acme for example is not so simple).

martes, 16 de agosto de 2011

Libninep

When this project was started one of the first questions that had to be solved was how to serve the 9P protocol from unix. We already depend on some Inferno libraries and Inferno includes a library to write 9P servers, libstyx. This library looks like an obvious choice. However, libstyx has some limitations: it does not support out-of-order replies, does not give the option to select on extra fds (we need that for the connection with X) and its representation of file trees is somewhat static.

Libninep is a modified libstyx, which tries to solve these issues.

Most of the content of styx(10) and styxserver(10) still applies to libninep (after s/styx/ninep/g, of course). The most important differences are explained in the next paragraphs.

Main loop
The loop of a libstyx application has this form:
for(;;)
  styxwait(&s);
  styxproccess(&s);
}
After styxwait returns, styxproccess reads the next request, looks for the corresponding function in s->ops, runs it if found, and replies. With libninep we have more options. The main loop of devwsys looks like this:
for(;;)
  nineplisten(&s, xfd);
  ninepwait(&s);
  if(ninepready(&s, xfd)
    xevent();
  ninepdefault(&s);
  updaterefs();
  ninepreply(&s);
}
There are some new library functions here: nineplisten adds a fd to the fd set to select from and ninepready checks if it is ready. The styxprocess function has been split in two: ninepdefault and ninepreply. However, the key difference is ninepwait, which is doing more than styxwait.

While styxwait just waits for a fd to be ready and let styxprocess to do all the work, ninepwait will read the next 9P request when possible, setting s->curc and s->fcall. ninepdefault will process this request, using s->ops when available, and probably turning a Tmessage into a Rmessage. Only when ninepreply is called this request is actually replied. This way, we can use ninepdefault for common operations but, if that is not enough, we can also process 9P requests on our own, just filling the fields of s->fcall and calling ninepreply when we are done.

What devwsys is doing is calling updaterefs after ninepdefault. This function will take care of updating the number of references to a window after an attach, a walk, or a clunk transaction. To do this with libstyx, we would have to give up on using its facilities to handle file trees and implement walks and dirreads ourselves.

Out of order replies
When reading event files (cons, mouse) the file server cannot reply immediately to a request, it has to wait until an event is received. This functionality is provided by two functions: ninepreplylater will put a request on hold and returns a pointer to a Pending struct, which can be passed to ninepcompleted to reply when we are done. After calling ninepreplylater, ninepreply will have no effect until ninepwait is called again.

It is the responsibility of the client to keep a queue of pending requests if desired. However, libninep will take care of marking them as flushed if a Tflush request is made.

File trees: binding
Libstyx includes some functions to help with the creation of file trees: styxadddir and styxaddfile. The advantage of using these functions is that using them we do not have to worry about implementing walks or directory reads, but they do not allow us to give several entry points to a given file. For example, devwsys has to give access to the files of a window in the wsys directory of every other window and on its root, and the clients of draw should be accessible by every window.

This problem is solved in Linux using links. The same problem is solved by Plan 9 (and Inferno) in a more elegant way by bind. So, libninep includes a ninepbind function. Its usage is quite simple: given the qid of two directories (pqid and qid) the files of qid will also be accessible from pqid.

In devwsys, the file system root only includes two directories: /wsys and /draw:
ninepadddir(s, Qroot, Qwsys, "wsys", 0555, eve);
ninepadddir(s, Qroot, Qdraw, "draw", 0555, eve);
When a new window is created (due to an attach request), files for that window are created inside the corresponding /wsys/n/ directory:
p = PATH(w->id, 0);
ninepadddir(s, Qwsys, p, name, 0555, eve);
ninepaddfile(s, p, p|Qcons, "cons", 0666, eve);
ninepaddfile(s, p, p|Qconsctl, "consctl", 0222, eve);
...
ninepadddir(s, p, p|Qwsys, "wsys", 0555, eve);
ninepadddir(s, p, p|Qdraw, "draw", 0555, eve);
ninepaddfile(s, p|Qdraw, p|Qnew, "new", 0666, eve);
Then, we bind the directories /wsys and /draw to the corresponding directories of the window:
ninepbind(s, p|Qwsys, Qwsys);
ninepbind(s, p|Qdraw, Qdraw); 
When a new client attachs to devwsys, wsysattach (which is server->ops->attach) will be called. This functions sets the qid to be returned to be the one corresponding to /wsys/n and the binding magic will take care automatically of walking and reading directories.
Note 1: rio's fs lets you to walk to /wsys/n/wsys, but no further. Devwsys, which uses libninep and bind directories, does not have this limit. You can walk as deep as you want into /wsys/n1/wsys/n2/wsys/n1/wsys/...
Note 2: The files of every draw client are added to /draw/n. Although /draw is bound to /wsys/n/draw we still have access to the /wsys/n/draw/new file which corresponds to the window.
Announce on unix addresses
You can only post your 9P server to a port using libstyx. Instead, libninep can also listen on unix!... addresses, as libixp does. A new function, ninepnamespace, gives the default namespace to use, maybe read from the environment variable NAMESPACE, else of the form /tmp/ns.$USER.$DISPLAY/ (as p9p and libixp do).

lunes, 18 de julio de 2011

Running devwsys

The most straightforward way to use devwsys is from 9vx. First, devwsys have to be launched:

user@unix$ devwsys
This will post a 9p server in /tmp/ns.user.:0/wsys (where user is the user name in the unix system and :0 is the current X display). Now, this 9p server can be mounted from 9vx to create new windows. An easy way to access devwsys is setting the wsys environment variable. Then, you will be able to open programs in new windows using window(1) with the -m flag:

term% wsys='#Z/tmp/ns.user.:0/wsys'

term% window -m acme
For a better experience the wctl rc script, included in the util directory of the devwsys distribution, can be used to provide a wctl service. Then, the -m option is not needed any more, new applications will use the namespace of the wctl process:

term% wctl=`{wctl}

post...

term% window acme
Similar commands will work from drawterm(8) too. Inferno, instead, is a bit different. Some files have different names, and cursors have a slight different format. To solve this issue, devwsys will deduce what cursor format is being used depending on what is written to the cursor file, while extra keyboard and pointer files are equivalent to cons and mouse. So, the only required step is to mount devwsys before launching wm(1):

; mount -Ab '#U*/tmp/ns.user.:0/wsys' /dev

; wm/wm
Have fun!

lunes, 4 de julio de 2011

Drawing cats

video

The main function of the drawing device is to draw the catclock window. This video shows how devwsys can be used for this fundamental task, and even to play sudokus, using 9vx on top of linux.

There are still many thing to be done: resizes are not working, programs segfault when the right mouse button is pressed, anything using fonts will fail and memory is being leaked all over the place. But we can draw cats, everything should get easier now.

domingo, 26 de junio de 2011

Bye bye drawfcall

Devwsys no longer uses an internal Wsysmsg representation. As a matter of fact, drawfcall support has been completely removed. Getting rid of this level of indirection devwsys can become a simpler program. It will only support one interface: a 9p file system.

The drawfcall infrastructure in p9p's devdraw has been verey helpful to get started, but it was starting to get in the way. We want to run devwsys in different environments, any unnecessary complication should be avoided. Mixing the draw and wsys file systems is complex enough already.

lunes, 20 de junio de 2011

Preview release of devwsys

A preliminary version of devwsys is publicly available:
This version does not have a functional drawing device yet, but can be used to create windows and read input events. Libixp and mk are the only external dependencies required to build. The Inferno drawing libraries are included in the repo.

Running devwsys a 9p server is started (by default in /tmp/ns.$USER.$DISPLAY/wsys). When mounted, a new window is created in the X display, and the following files can be accessed:
  • winid: an unique window id.
  • label: read or write it to get or set the label of the window.
  • mouse: to get mouse and resize events.
  • cons: to get keyboard events. Alt is the compose key.
  • consctl is only a dummy file. cons is always on rawon mode.
  • wsys/n/ gives access to the file system of all the open windows.
  • draw/ and draw/new are not functional yet.
Devwsys still needs much more work, but is progressing. There are still many aspects to improve, and more files need to be implemented for rio(4) compatibility. The drawing device is the top priority now.

Libdrawp release

The libdrawp repository is now publicly accessible:
For more information about libdrawp, see this post. Let me know if you find the libraries useful or have any problem with them.

jueves, 16 de junio de 2011

Devwsys

Devwsys will be a new program. Its main components will be some libraries from Inferno (libdraw, libmemdraw, libmemlayer and also lib9 and some pieces from other libraries), libixp and a considerable amount of p9p's devdraw code, as well as from the Inferno kernel devices.

The objective of devwsys is simple in its conception: serve a 9p file system similar to that one of rio(4), with the difference that new windows are not created in a rio instance, but in the windowing system of the host (X11). In fact, a devwsys window will be more limited than a rio window, since it will not include any text editing capabilities, they will only be graphical windows (eg. consctl will only be a dummy file, cons will always be on rawon mode).

In the future, it could be convenient to use an alternative server, for example to work with the p9p's devdraw protocol in stdin/out or to replace libixp with something else. In a similar way, although it is not part of this project, it should be possible to use other windowing systems (like OSX, Windows or, if it takes off, Wayland). In order to achieve this goal, devwsys will use the internal message representation of devdraw. 9p messages will be converted into Wsysmsg's and a runmsg function will take care of interpreting this message and perform the relevant (window system agnostic) function calls. If we want to replace libixp we will only need to build the corresponding Wsysmsg from any form the requests take and supply a reply function, and if we want to support other windowing system only the functions used by runmsg will have to be implemented.

Of course, we are not there yet, but are getting a bit closer with every line of code. A very preliminary version of devwsys will be made public at the end of this week, with the basic functionality to create new windows and some support for input events (mouse and cons files), but not drawing yet.

martes, 14 de junio de 2011

(Not so) portable drawing devices

The quick results obtained with the drawing libraries led me to think that it could be possible to port one of the existing devices without much work. I was wrong.

The devices in 9vx and Inferno are kernel devices and they make use of all the facilities a real kernel gives you. In particular, they use kprocs and chans, which are not obvious to get working in an Unix environment (not without having to add too much code). There is a kproc implementation based in pthreads I have not got my hands on yet, but anyway we want to stay away from threads, because multi-threaded programs and X11 do not play well together. Extracting only the useful bits from these devices is certainly doable, but not an easy job (not particularly difficult, but is not the kind of thing you can try in an afternoon).

Plan9port's device, however, is not a kernel device. It is an external program. It looks like that could make things easier but, in practice, not so much. I have tried but, after writing some sed and ed I realized the task was much larger than I had originally thought (or, should I say, hoped).

So, what is the plan for the wsys device? Writing our own. However, we have nice examples available, so it is not a rewrite from scratch, not at all. Some portions can be taken from p9p, some other portions from Inferno or 9vx. Surely some parts will have to be reworked, refactored or totally rewritten. The idea is to start with something very basic and add features one at a time, always having something we can compile and test. This way, not only we have a better control of the code, it will also help me to understand better how it works and all the involved problems. It will take some time but I hope the result will be worth all that work and, to be honest, being able to compile one of the existing devices with only a few changes would have been too good to be true.

For the moment, this was the last experiment. The following posts will discuss the development of the new device.

lunes, 13 de junio de 2011

Portable drawing libraries

A drawing device needs some Plan 9 libraries: draw(2), memdraw(2) and memlayer(2). These libraries have been ported to Unix as part of p9p, Inferno and drawterm. Another option, used by 9vx, is to use the original Plan 9 libraries.

The C code in Plan 9 is not ANSI C, so it cannot be compiled without modification with standard compilers (although gcc has support for the extended features of Plan 9 in its latest versions). Therefore, the .c files in 9vx are not exactly the same ones that are found in /sys/src. However, the changes are few enough to be easily applied as a set of .ed files.

If you try to autogen the .c files in 9vx from current Plan 9 sources (for example, from sysfromiso) you won't get too far. Unfortunately, those scripts have not been kept up to date with the latest changes in sources. Another problem is that some of the last bugfixes have not been included in the Plan 9 distribution (yet?).

As part of the experiments performed to find a version of the drawing libraries suitable for the wsys device a repository with portable versions of libdraw, libmemdraw and libmemlayer has been set up in bitbucket. The .ed files and the autogen scripts have been updated to work with the latest versions (from the beginning of June) and other new bugfixes are applied as additional ed patches (only memdraw32bit, actually). Make, and not mk(1), is used to build the libraries.

The libdrawp repository (private for the moment) will be made public soon, just in case somebody is interested. It just needs a README and the LPL license.

However, these libraries will not be used in the wsys device. The libraries in Inferno are not going anywhere any time soon (they are needed for the native implementations and in other host environments), they are ANSI C, the build process uses mk(1) and include some bugfixes not present in Plan 9. That's what we are using for the wsys device. I don't think I'm going to bother maintaining libdrawp in the future, but it was another interesting experiment.

9win

9win was the first of a series of experiments. It consisted on adding a 9p fileserver to p9p's devdraw using libixp.

The whole file system required by a wsys device was not implemented. In fact, only the most basic functionality made it into 9win. An attach created a new window with an unique id and a mouse event caused a message to be printed (to stdout, not even using the 9p file system!). Although 9win was just  small hack, I consider the experiment quite successful: it showed that libixp was a good candidate to get the project running (certainly faster than writing our own 9p implementation), that 9p can be served and X events read without using dedicated threads (9win was not a multi-threaded program), and that p9p's devdraw design allowed to be easily extended to support multiple windows and other protocols.

A devdraw device which servers 9p is basicly what we are after, but in order to compile 9win you needed p9p, and p9p is not an acceptable dependency for 9vx, drawterm or Inferno. That's why the next series of experiments consisted on looking for a version of the libraries which was portable enough for our purposses.

GSoC2011: Unification of X11 code and wsys device

The main objective of this project is to reduce the duplication of code between the different Plan 9-like drawing devices which run on X11. Currently, there are four similar (but subtly different) implementations. Charles Forsyth (mentor of this project) made a very clear explanation of which is the current situation, I've taken the freedom to reproduce it here:
  1. The existing devdraw program runs on x11 and osx. It implements the plan 9 graphics protocol, but embeds that in a novel non-9p protocol. Only p9p's programs can talk to it, through a modified libdraw.
  2. Inferno and plan 9 use the original libdraw (more or less) to access draw's primitives through a name space that's implemented by draw(3), a kernel driver. draw(3) uses libmemdraw for graphics. Applications access the name space by binding a local device or mounting a remote 9p connection.
    1. Plan 9's kernel driver port/devdraw.c gets */screen.c to do the work, typically in flushmemscreen (or hwdraw). 
    2. Inferno's kernel driver uses x11, osx, windows and native variants of win.c or screen.c (roughly speaking), with most of the work done in attachscreen and flushmemscreen.
  3. Drawterm and 9vx implement draw(3), drawing using a libmemdraw that devolves a lot of the work to special x11 and osx variants, and using a few geometry primitives from libdraw. 9vx applications are just plan 9 applications, and they use plan 9's libdraw.
Except for minor differences that can get in the way in practice, Inferno, Plan 9, and 9vx can all draw on each other's screens. They can also all draw on a drawterm screen (again, if not prevented from doing that by minor differences).
Inferno and drawterm are the only ones that run on Windows. The others, except for Plan 9, run on Unix-y systems like Linux, and on MacOSX.
Only p9p programs can talk to the devdraw program directly, and (I think this is right) they can't talk to any of the other systems directly. Generally, p9p is the odd one out. On the other hand, p9p uses a separate program to do the graphics, and that has advantages in hosted environments.
In order to improve this state of things, a new wsys device will be written. This device will be a unix program which uses xlib and serves 9p. Being a 9p server, it will be possible to mount it from Inferno, 9vx and drawterm without any modificaions. It is still to be decided how the new device will be used from p9p.