Prerequisite: A simplified outline of machine learning -- The goal: Given a (relatively small) set of data (the training set) consisting of input and output, train the computer to correctly predict output for new, unseen inputs. People design algorithms and write programs that, based on the training set, generate models that do the actual prediction. By convention, inputs come in the form of feature vectors, and all possible inputs can be expressed through different vectors in a vector space (the feature space). Machine learning has a plethora of applications in many fields including computer vision, biology, advertising, and music. Our work focuses on musical applications of machine learning (although much of our work can be applied to other fields as well).
If you want to learn more about machine learning, this seems like a good place to start.
Our work is highly based on the Wekinator, so checking that out would be useful as well. In summary, the Wekinator is a tool that allows users unfamiliar with machine learning to use powerful machine learning algorithms to generate models that can, among other things, drive sound synthesis.
Interactive supervised learning, as used with the Wekinator, is an iterative process where the user incrementally learns about the feature space as the user evaluates the model through realtime playback. Composers using Wekinator to design digital instruments have often noted that getting feedback from the computer is an important part of the creative process. In fact, sometimes the computer generates models that respond to input in unexpected, exciting ways, inspiring users to change their designs for the better. Motivated by this, we explore different techniques that will empower the computer to communicate even more useful and interesting feedback to the user.
In the Wekinator, the (slightly simplified) procedure to train a model is to (1) Specify the input devices (2) Specify the number of output features (3) Demonstrate training data that maps input to output values and (4) Train a model and evaluate it through realtime playback. Notice how much information the user must have in order to begin the design process in Wekinator. This procedure assumes that the user has a preconceived notion of how the instrument should be played, i.e. which playable motions should control which parameters of the resulting sound. The composers who used Wekinator were most satisfied with the software when they learned that their preconceived notions were WRONG, and the Wekinator showed them a different possible mapping which resulted in a more natural and/or expressive instrument.
Intuitively, we should move away from this assumption that the user knows precisely how input should map to output. After all, imagining model playback as a rapid prototyping environment highlights the ideology of Wekinator as a meta-instrument (in other words, thinking of the Wekinator as a tool used to design digital instruments (although it must be noted that the Wekinator can be useful in many different domains, some non-musical)), in which case we should assume the opposite, that the user has no idea how to design the instrument and must explore many possibilities before coming to any design decisions!
Our work is to design a procedure for interactive supervised learning where the user can train a model without specifying a mapping from input to output. Instead, the user will simply demonstrate input behaviors and possible outputs, leaving out information about how these two should relate to each other. The computer will then decide which inputs should map to which outputs, and generate a model. Essentially, we want to remove step number 3 in the Wekinator procedure, or at least make it an optional step. Since the interactive supervised learning process relies on iteration, we plan to allow users to explore multiple models with little effort. In addition, we want to allow the user to provide certain constraints (ex. "I want THESE inputs to map to THESE outputs, but i'm not sure about the rest of them") that can steer the computer towards generating more useful and interesting models.