Lisa Torrey
Summer 2002

This report summarizes two projects: the design of a Java interface to the 3D Scene Generator, and the integration of scenes and images into the Foundational Model Explorer.


Scene Generator Interface

The graphics server is an application written in Skandha that can construct 3D scenes out of 3D anatomical model primitives. The Scene Generator is a Lisp server that takes commands to create a scene, tells the graphics server to construct the scene, renders it internally, and saves a 2D snapshot to be retrieved for display. Commands to create and manipulate scenes first come from user interfaces to the Scene Generator.

There is a Perl interface to the Scene Generator, called the DSG, on the SIG demo site (most recently the responsibility of Kevin Hinshaw). There is also a partial Java applet interface written in 2000 by Bruce Campbell. I wrote two full Java applet interfaces, one that uses the Swing package and one that doesnít, using those two programs as models. Both require jdk1.3.1 for the layout to look right, and the Swing version requires it to function at all.

Both the interfaces and the Scene Generator often communicate with a Foundational Model server, which answers queries about structures and their relationships. Currently, all versions work from an FM server that runs at tela 2322.

Operations

The operations on structures are Add, Remove, and Highlight. Adding a structure adds its 3D model (if it has one) to whatever is already in the scene, and traverses a specified hierarchy to add any of its descendants that have models. Removing a structure removes its model and those of its descendants. Highlighting a structure doesnít change that structure, but makes every other structure mostly transparent, leaving it visible regardless of its depth in the scene.

The operations on the scene itself are Re-center, Rotate, and Zoom. Re-center orients the scene back along its original axis and sets the zoom distance so that everything fits in the display. Rotate turns the entire scene by a specified amount along an axis in one of these directions: clockwise, counterclockwise, right, left, up, down. (The icons in the Swing version illustrate fairly well what these directions mean.) Zoom moves the viewpoint in or out by a set amount.

The file operations are Save and Load. You can save a file as an independent scene, as a partial scene that can be added to another (an ìAdd-onî), or as a VRML file. You can load any scene or add-on that has been saved.

Interface Design

I focused on the following goals for the design: intuitive function and command duplication.

Because I found the DSG difficult to work with at first, I wanted the applet interface to function more intuitively to a new user. I designed it to look like a drawing program, with a ëtoolbarí of buttons clustered on the left and a ëcanvasí on the right. I tried to choose button names that described their functions in general graphics terms, and I replaced a text button with an icon wherever possible (in the Swing version). Since this kind of graphical editor interface is familiar to most people, I felt that users would be more comfortable with it. I was also careful to include understandable error messages, and to prevent users from making commands that might put the server in an unpredictable state.

The complicated nature of the dataó anatomical structures and their partsó suggested that a user might need several ways of giving a command. For example, suppose a user wants to highlight something, but he only knows its location in the scene and not its name. Or suppose the opposite; he knows its name but not its location. The study of anatomy encompasses so much raw data that either situation is likely. In order to be useful over a broad range of situations, the interface needed to support duplicate ways of giving commands.

Code
I focused on making the code portable, friendly to change, and reasonably efficient.

Following Bruce Campbellís example, I made each operation a separate class, an instance of a superclass that contained shared information. Another set of classes controlled the display, and a driver class contained most of the interface. Without too much difficulty, the operation classes could be extracted from the parts of the code that are dependent upon the particular interface, making the code portable to other applications. (The later JSP versions turned out to be even less interface-dependent; those are the ones Iíd actually suggest using.) This setup also made changes fairly simple, because any one operation could be modified without affecting another.

Efficiency was mostly limited by the graphics processing, but some of the bookkeeping that I introduced to implement parts lists (described later) grew fairly complicated. I organized the structures in the scene into nodes in a linked list, and gave each structure node a linked list of parts.

On several occasions, I needed to perform file operations on the UNIX system that were not possible from within the applet. To get around these limitations, I created two CGI scripts and used one that Bruce Campbell had created.

The operation classes process user input and make calls through the SkandhaBox object to manipulate scenes. Here are descriptions of other classes, and of CGI scripts:

1) SkandhaButton: an abstract superclass that contains common methods and data for the subclass buttons, of which there is one for each operation.
2) if_termserv: a connection to the DSG server, currently on quad at port 63070.
3) SkandhaBox: sends commands to the server connection, and contains the output display and the structure list.
4) SkandhaOutput: the output display that loads and shows images.
5) LinkedList: the list of structures in the scene (contains Structure objects, which in turn contain Part objects).
6) Starter: the applet itself, and the interface that the user interacts with.
7) convert.cgi: converts the 2D snapshot from tiff format to jpeg format.
8) replace.cgi: prepares a file to store a structure list.
9) saveList.cgi: stores a structure list in a file.

Structure and part lists

I decided early on that maintaining a list of structures and their parts would serve several purposes. It would allow users to find and select terms without having to type in long names, to see the relationships of terms to each other, and even to operate on more than one term at a time. It would also prevent them from trying to operate on terms that werenít present in the scene. When a user knows the structure name, but not its location, the structure list is quite useful. When she knows the location and not the name, she can click the location on the image to retrieve a part label from the server (a pre-existing function). The applet then provides new buttons that allow her to operate directly on the selected part. The only change I made to this capability was to coordinate these operations with the structure list.

Maintaining the list, and later coordinating it with saving and loading scenes, proved to be some of the most complicated aspects of the program. In most cases, it required information directly from the foundational model server.

When I added a term, I needed to ask whether it was a descendant of a structure that was already present, and if so, to place it in the part list of that structure instead of treating it as a new structure. A ëget-ancestorsí query lists the possibilities, and the program checks to see if any of them are in the structure list. Then, I needed the names of the descendantsó but not all of them; only those with models, which meant that a direct query to the FM server would not work. In fact, the only way I found to accomplish this was to issue a ëshowí command, which returned the name of every model in the current scene, and to process the string to get the parts of the most recently added structure. The DSG supposedly has a method of determining which terms have models, but itís not a correct method, because there are many more models than it suggests. When I removed a term, I needed to search the linked list to find it. If it was in a descendant list, because such lists may contain more than one level, I needed to find any of its descendants that might also be in that list, using a ëget-descendantsí query.

None of these maintenance methods are perfect. They are highly dependent upon the syntax of responses from the FM server and the graphics server, and all sorts of String exceptions are sure to occur in the future with changes in syntax. A more reliable way to maintain lists would be to build such functions directly into the graphics server. At the moment, the applets arenít throwing any exceptions, but there are occasional issues caused by inconsistencies in the FM server. If a supposedly reciprocal relationship is not perfectly reciprocal (i.e., term A is a descendant of term B in one hierarchy, but in the reciprocal hierarchy, term B is not a descendant of term A) then the lists will behave incorrectly.

Saving/loading

Structure lists needed to be saved along with the scene files, because loading a file without one wouldnít be very useful. In order to do that, I used the CGI scripts described above to save a file with the extension ë.listí that had the same name as the scene. (I used scripts because Java applets canít write to files without having permission problems.) I also saved the highlighting state of the scene in those files; although the loaded image would display the correct highlighting, the interface wouldnít know that anything was highlighted.

The graphics server saves scenes as ë.strí files in ~hinshaw/public_html/dsg-thorax/state, by default, but in these applications I have set the files to go instead to ~ltorrey/public_html/dsg/scenesdir. The ë.listí files go to the same place, as do VRML files. (The snapshot image files currently go to ~ltorrey/public_html/dsg/results.) Those locations can easily be changed by modifying strings in the SkandhaButton class.

Future Improvements

There isnít a real database for the 3D model primitives; at the moment, the graphics server uses a file called ëmodeldbí instead. A real database certainly could be useful. And of course, more models could be added.

The graphics server itself could use some more functions. As I mentioned before, functions to maintain the parts lists would significantly simplify an important part of these programs. I would also suggest a function that creates an IML file from a scene, so that a user could choose to add the scene to the image database as an annotated image.

Currently, image and scene files (in my public_html/dsg folder under ëresultsí and ëscenesdirí, respectively, and also in public_html/manager) simply build up in their directories. Especially in the results directory, this takes up a lot of space before long. I didnít find a good way to automatically delete files, because there could be more than one user at a time, and you can never really know what files may be in use.

The scene files could ideally be saved to usersí computers instead of a directory on quad. Then the program could offer them a list of scenes to load, instead of requiring that they remember the name of a saved scene, and they wouldnít be able to use or over-write other usersí scenes. Also, coloring doesnít always carry over correctly when you save and load a sceneó the heart, in particular, loses its contrast and becomes all red.

The new FM server is ready to be used at quad 8098, and once the scene generator uses it, the applets could be modified to use it.

The parts lists could be structured more, to reflect multiple levels in the hierarchy. This could get quite complicated, although it would be nice to have. The lists could also include all descendants, indicating ones with models with some sort of flag, but a list like that might turn out to be unmanageably long for a user to work with.


Foundational Model Explorer Integration

The FM Explorer is a text-based tool for browsing hierarchies of the FM. It was one of the candidates for integration of scenes, images, and text, and it was chosen because itís under current development. The program was already implemented using Java Server Pages, so the Java classes I had written were easily transferable to it. Instead of an AWT-based interface, all user input had to come from an HTML form.

The main decision to make was the style of the integration. A user would need to be able to navigate back and forth between the three kinds of information easily, but in a limited amount of space. The FME began with two frames: on the left, a hierarchy, and on the right, a content page that contained information about the selected term in the hierarchy. I changed the content page into another pair of frames: on the top, a ëtabí for each type of information, and on the bottom, the chosen information.

Once the layout was acceptable, there were decisions to make about how the different pages of information would correspond. They should be tied together so that a change in one page is reflected in the other pages, but the link should not be so strict that the users' explorations are limited. I decided to put in place some default connections, like displaying default images and adding a 3D model by default for each content page visited. Beyond those defaults I decided to allow some freedoms, like adding arbitrary structures to scenes without necessarily changing the rest of the settings.

Implementation

Selecting a concept in the hierarchy on the left now brings up four ëtabsí on the right. The first is the original content page, indicated by the concept name. The second is an image page, which displays a default set of ëthumbnailí images retrieved by the image manager, using the concept name as a query. The third is a list of all the concepts that appear in the current content page; they can be selected singly or in groups for easy addition to the scene. The fourth is the interface to the scene generator, which automatically and cumulatively adds the current concept name (if it has the model for it), allowing anything already in the scene to remain.

Selecting a different concept, either from the hierarchy or the content page, changes the set of images and attempts to add the new concept to the scene. Selecting one of the small retrieved images zooms in to a larger version. Returning to the small images, another link provides a search interface for images. Images of a structure are retrieved without changing any pages, but once they have been retrieved a button can be used to ëexploreí that concept, changing the content, image, and scene pages accordingly. From the scene page, a part can be selected by left-clicking on the display, and a similar button appears to ëexploreí the selected part. Any structure, though, can be added to the scene without immediately changing the other pages.

Scene section

It turned out to be possible to make the interface look remarkably similar to the original applet one. The HTML layout and the user-initiated function calls are in the file ëportal.jspí, and the Starter class receives those function calls and passes them to the appropriate class for handling. The SceneList class creates and displays the list of concepts from the content page.

While user input in the applet comes from pop-up frames, input to the JSP versions comes from forms that appear beneath the main interface. However, all the possible operations, and the ways that theyíre carried out, are essentially the same as in the applet versions. I made two JSP versions with independent (though often redundant) code: one standalone and one linked with the FME. The standalone one could be more easily modified for other applications.

Image section

Rati Sharma designed a Java interface to the Image Manager, and I used her work to integrate images into the FME. Her original code came in one file, but I separated it into several classes so that the integration would be easier:

1) ImageManager: communicates with the HTML interface.
2) Action: controls the image retrieval process and displays the first image.
3) NextAction: displays the next image that was retrieved.
4) PreviousAction: displays the previous image that was retrieved.
5) GetImage: performs some of the image retrieval functions.
6) Stemmer: stems words in the query before the search begins.
7) Thumbnails: communicates with HTML image page to display small images.

I also used one CGI script, moveImg.cgi, to move a retrieved image from wherever it originates to my ~/public_html/manager directory. (They had to be in a public folder to be displayed in HTML.) As a result, image files build up in this directory, too.

The interface had to have access to the image database and to the mysql class package. Rati ran her original program from sulcus and included the mysql package from her home directory in the classpath, but these options werenít available for a JSP version. So, jdbc:mysql://sulcus.biostr.washington.edu/image_repo is the full location of the database, and it now accepts remote logins so that it can be accessed from the web. On the machine that eventually serves the FME, the mysql package will need to be included in the Java classpath.

Future improvements

The image section needs a lot of work. It retrieves the same image multiple times (with different annotations), and takes quite a long time to do so. It might be useful to limit the number of retrieved images on the thumbnail page, so that itís faster. Also, frequent array errors and occasional out-of-memory errors happen during image retrieval.

On the zoomed-in versions of the images, a user should be able to click to get the label of a region. In the same manner as in the scene, a button could then be made available to explore the concept corresponding to the label.

The structure list could be generated more efficiently from the Java code for the content page rather than from the HTML. If we could implement a more direct way to easily add terms from the content page to the scene, the structure list could be removed altogether. At the moment, the terms on the content page are links to other content pages, but perhaps small links beside the terms could serve a purpose like this one.

The JSP versions can be updated to the new FM server in the same way as the applet versions. (See local documentation files for detailed information.)