ISI Co-authorship pipeline: Make loading cause ISI files to be cleaned.

A subtask of ISI Co-authorship network pipeline

Stage:
Requirements Specification

Description:
When the user loads an ISI file, that ISI file should immediately be cleaned, and the data manager should show the original and cleaned ISI file in the following format.

Original Uncleaned ISI File
     (child) Cleaned ISI File

Next Step:

Technical Design (Skipping feedback on requirements for now)

Stage:
Requesting Input on Technical Design

Description:

The straightforward way to implement this would be to alter ISI's validator to validate (as usual), then call the isi->prefuse.data.Table converter,and then call the isi duplicate remover.

However, implementing this exactly as it is described would break how NWB is expected to handle the loading of files. As you know, when a file is loaded in NWB the only "converter" called is that file type's validator, in this case "PrefuseIsiValidation?" for the type file-ext:isi. In order to automatically clean the ISI file the validator would additionally have to perform a conversion (file:text/isi -> prefuse.data.Table), and then run a mutating algorithm on that file (removing duplicates). This is problematic for a few reasons. It breaks our model of what a validator should do, and forces us to convert to prefuse.data.Table, and to clean that table, either of which we may not want to do in future workflows. It also is obnoxious from a coding perspective, since we would have to entangle the validator with the conversion and cleaning algorithms, only to have to potentially untangle them at some later point.

A possible way around this that Bonnie and I discussed would be to have a separate specialized load option in the load menu, maybe called "Load and Clean ISI File". If the user selected this option then the file would be both loaded and cleaned. If the user selected the normal load option, the file would simply be loaded as an ISI, and the user could still choose to manually invoke the cleaner (which might appear in the preprocessing menu).

Next Step:
Discuss Technical Design Possibilities

Stage: Design implemented and user feedback received

Description: Automatic ISI loading and cleaning was implemented as described above, in the plugin edu.iu.nwb.composite.isiloadandclean. Katy was shown a demo, and approved the implementation after a few minor changes.

Next Step: Documentation

This has been done for a while. Closing.