Requirements Specification:

This feature should cover everything in the following two tickets

Improved Save Functionality (ticket:160)

Need cleanup and persistence capabilities for Data (ticket:32)

The Plan (v2.0):

1) Overview:

Users will be able to save and load "Sessions", which persist all the data needed to pick up from where a user left off when they last closed NWB. This persisted information currently includes 1) Each data item in the data manager, and its associated metadata. 2) The contents of the console. Loading a previously saved session will put all the data from the previous session into the data manager, and fill the console with the persisted console contents.

Value to Users

Basically, sessions should make it easier to "live" in NWB. For frequent users, having to load in all the data you are working on each time you start NWB is quite annoying. Allowing NWB to maintain its state reinforces our goal of making it the center of a Network Scientists workflow. If, after using NWB to process some data, the data must be saved out to the regular file system, there is no incentive for that user to continue processing their file in NWB as oppose to some other tool on their desktop. But if their data already magically appears when they load NWB, they will be encouraged to continue working in NWB. In breaking down barriers to regular use, we make it easier to use for research, and making it more enticing for users to contribute their own algorithms.

After writing the paragraph abovee, it occurred to me that always persisting all the data in the data manager may lead to a build-up of data that the user does not care about, forcing them to manually remove old data items. Further compounding this problem, you cannot currently remove a data item without removing its children, so it is not possible to remove the intermediate steps from starting data to a final result without removing the final result itself. We should put some serious thought into this issue.

2) User perspective:

The user will interact with the session feature in the following ways:

  • In the file menu, users can "Save Session", "Save Session as...", or "Load Session".
  • If a user attempts to quit NWB without having saved their session, they will be prompted to save their session before quiting.
  • When a user starts NWB, the most recent saved session will be restored automatically (maybe) (We may need to think more about how this will work (how/if we want to restore a session when a user starts NWB up again). Eclipse always pops up a little dialog to choose workspace before the main GUI even starts. What if the user wants to start a completely new session, and wants to ignore their old session, even if they saved it from before?).

    From Bonnie: Restoring previous session is a nice feature but should not be a default setup. Down the road, we should allow users to specify through the preference service whether the tool will restore the previous session. Behavior Overview:

    • Save Session as... menu option: A special dialog pops up (described later). The session is saved with a specified name in a specified location, where each data item is saved into a specified file format.
    • Save Session menu option: No dialog pops up. We save the session into a default location, overwriting any session that existed their previously, using default file formats for all data files. (Defaults are determined with some smart guessing, built-in defaults, or possibly with preference from the preference service eventually).
    • Load Session: Prompts for if you want to save your current session. Allows you to choose a previously saved session with a standard file load dialog. Restores the session.
    • Prompt on NWB Close: If yes, execute Save Session as... (or regular Save Session?).

Save Session As... Behavior Details: This is basically what Russell has up on the whiteboard in his room. When a user saves their session, a special save session dialog box will appear. The save session dialog box will list each data item currently in the data manager, with a drop down box next to each data item where the user can choose what format they would like to save the data out to. (The dialog will determine smart defaults based on which converter path is considered the most reliable. Alternatively, have preferences that specify the file formats that each in-memory format should save out to) Using a standard file browsing dialog, the user can either write their session over an old session, or can save the session with a different name, entirely separate from their old session.

Starting up: When a user starts Network Workbench, NWB will restore the most recently saved session, placing all the previous data into the data manager, and filling the console with the log from the saved session. Users can also manually specify a session to load once NWB is already running, using the "Load Session" menu choice. (Several issues related to this behavior are addressed in the User perspective bullet point above).

3) Implementation perspective:

Saving a new session:

When saving a session, we call getAllData() on the DataManagerService?. For each data item, we convert it to a output file format, and write it to disk, along with its metadata. We also get the console output (somehow), and save that as well. All the data files, metadata files, and the console log are saved in a directory (called "my session" or "session-02-01-2008" or whatever) inside the "sessions" folder in the nwb directory (by default). In the sessions directory we also have a hidden file which remembers which session was last saved, and where it is located (So that we can restore it by default later).

From Bruce: It is not necessary to have a hidden file since CIShell requires OSGi's PreferenceService. Just save the location of the last saved session in there.

Saving over an old session:

This will be the same as saving a new session, except 1) If there is an updated version of any file, we overwrite it with the newer version, and 2) if there is a saved data file which is not in the data manager currently, we delete it.

(Still need to flesh out how we determine if there is an updated version of a file. Seems like it might be a little complicated.)

Load a session:

The user will have specified the directory which represents the session they want to load. We load all our session data (data files, metadata files, console log) from here. We use the metadata files to remember what format the data files were originally in, and use the conversion service to convert them to that format. We also use the metadata files to determine which data item was a parent of which, so that the children and parents of data in the data manager will look the same as before. We write the contents of the console log to the console.

4) Design decisions:

Here are a decisions that went into formulating the above design. I thought it would be helpful to make these explicit here, rather than allow them to lay secretively between the lines of the above description.

Keep the concept of saving and concept of persisting sessions separate Originally we considered simply prompting the user to save (in the regular sense) every data item in the data manager when they attempt to exit. While this would certainly be easier to implement than the current design, several issues arose with attempting to restore the state of the data manager when NWB started up again. What if a user moves or deletes one of the data files? What if they save everything to a usb stick, which is then removed and can no longer be accessed by NWB on further sessions? Things would be bad enough if they saved it to a designated folder, let alone allowing them to save it anywhere on their computer. It was decided that mixing the user-space "save" folder (where users generally feel free to tinker with the contents) with the sessions persistence folder (which needs to be left alone in order to be properly restored) would cause confusion and management issues. So, at the cost of potentially have redundant saved data, and possibly confusing the user about the difference between saving files and saving sessions, we decide to keep them entirely separate. The issue of redundant saved data can probably not be helped, but hopefully we can keep the idea of saving a single piece of data separate from saving a session by using consistent terminology in menus, and keeping the saving and restoring of sessions as transparent as possible.

Use the conversion service to convert data to file formats, and back to the original in-memory format when saving a restoring sessions

While this may be a necessary choice, given no viable alternative, it is worth specifically mentioning because of the potentially major implications. The more transparent we make the saving and restoring of sessions, the more we are promising the user that the state of their data when it was saved to a session will be the same as the state of that data when it is loaded from a file converted back to its original format. The problem is that during the process of converting to a file and back, converts may be making minor (hopefully not major) alterations to the data. Certainly they will at least change ID numbers. The session persistence functionality will be a significant test of our conversion infrastructure. Hopefully we can mitigate some of our known flaws by making good default choices in what converter path we use to save in and save out in-memory data formats.