Question

I don't understand PS10. Why separate your data for each k in problem 1?

Answer 1

Did you read the "Background Overview" in the PDF file?

Answer 2

Yes, I have. I understand k-means clustering, but I don't know why the instructions say to separate your data each time. Are you meant to have 12different sets of data, or take the average of the splits, or something else?

Answer 3

I do not see where it says you need to "separate your data each time." What I see is it taking small chunks from \(25\le k \le 150\) where you are evaluating each to build up a list of errors to graph.

Answer 4

This is cut and pasted from the instructions:

Iterate over k in increments of 25 from 25 <= k <= 150 and for each k do the following:
1.Partition your data set into a training and holdout set, ...

Answer 5

Typically training and holdout are part of machine leanrning. Have the lectures covered any topics like that?

Answer 6

Machine learning has been covered, but I don't think training sets and holdout sets have been covered.

Answer 7

A couple rweferences that might help with the terms:

http://www.sussex.ac.uk/Users/christ/crs/ml/lec03a.html

http://www.astroml.org/sklearn_tutorial/general_concepts.html

That seem to be related to how it is being used?

Answer 8

That helped, but I still don't understand why it gets split 6 times.

Answer 9

From some I read, it looks like the program is intended to be able to be run on very large data sets. So some of the chopping things up may be to work on it in easier pieces to work with.

Answer 10

I think I understand it now. Thanks so much!