Publication Venues

• Main Neural Network Conferences– NIPS (Neural Information Processing Systems)– IJCNN (Intl Joint Conf on Neural Networks)

• Main Neural Network Journals– Neural Networks– Neural Computation– IEEE Transactions on Neural Networks

Publication Venues

• Main Machine Learning Conferences– ICML (Intl Conf on Machine Learning)– COLT (Computational Learning Theory)

• Main Machine Learning Journals– ML (Machine Learning)– JMLR (J. Machine Learning Research)– JAIR (J. Artificial Intelligence Research)

Underfit and Overfit

Need for Bias

22n Boolean function of n inputs

x1 x2 x3 Class Possible Consistent Function Hypotheses

0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1

1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1

1 1 0 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1

1 1 1 ? 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1

No Free Lunch

• Any inductive bias chosen will have equal accuracy compared to any other bias over all possible functions (assuming all functions are equally likely). If correct on some cases, must be incorrect on equally many cases.

• Is this a problem?

– Random vs. Regular

– Anti-Bias (even though regular)

– The “Interesting” Problems – subset of learnable?

Automatic Discover of Inductive Bias

• Defining the set of Interesting/Learnable problems – No Free Lunch concepts

• Defining the set of available inductive biaseso Proposing novel learning algorithms

o Analysis, comparison, and extension of current learning algorithms

o Defining/discovering a set of biases which covers I (Interesting problems)

o Parameter Free learning algorithms – Automatic selection of learning parameters

• Automatically fitting a bias to a problem – Overfit, underfit, noise issues, etc.

• Automatic Feature Selection

ADIB (Cont.)

• Dynamic Inductive Biaseso Pre-selection of an appropriate bias based on the application data set

o Automatically selecting a bias during learning

o Bias which adjusts dynamically in time during learning

o Bias which adjusts dynamically in space during learning (different parts of the problem space are better learned with different biases, including differing parameters in one bias).

o Combinations of the above

• Combination of Biaseso Linear and non-linear combinations of biases

o Dynamic combinations of biases

o Ensemble variants