(The below text version of the notes is for search purposes and convenience. See the PDF version for proper formatting such as bold, italics, etc., and graphics where applicable. Copyright: 2022 Retraice, Inc.)
Re99: Math and Code, Bottom-Up and Top-Down
retraice.com
Just-in-case and just-in-time as strategies for what to learn when.
What has become obvious while focusing on the math and code of AIMA4e; needs, constraints, real goals, collapse into specialization; just-in-case (JIC) or bottom-up, and just-in-time (JIT) or top-down approaches to learning and doing; pinching knowledge by searching from opposing directions; evidence of math and code needs; remembering purposes.
Air date: Wednesday, 28th Dec. 2022, 10:00 PM Eastern/US.
Now evident, after the December to Remember Math and Code Event
o the math and code won't stop being a major part of studying AIMA4e;^1 o constraints are severe, especially time, so we'll teach to the online exercises early instead of late in each of the six periods; o AIMA is not enough, and not a goal; o the realness is outside, not in books; o we've collapsed into specialization in the most general discipline (AI); o the body of math and code knowledge is vast and intimidating, but so are cities, for which the best strategy is to explore and enjoy.
Just-in-case, bottom-up and
just-in-time, top-down learning
Deisenroth et al. (2020) give this explanation:
We can consider two strategies for understanding the mathematics for machine learning:
Bottom-up: Building up the concepts from foundational to more advanced. This is often the preferred approach in more technical fields, such as mathematics. This strategy has the advantage that the reader at all times is able to rely on their previously learned concepts. Unfortunately, for a practitioner many of the foundational concepts are not particularly interesting by themselves, and the lack of motivation means that most foundational definitions are quickly forgotten.
Top-down: Drilling down from practical needs to more basic requirements. This goal-driven approach has the advantage that the readers know at all times why they need to work on a particular concept, and there is a clear path of required knowledge. The downside of this strategy is that the knowledge is built on potentially shaky foundations, and the readers have to remember a set of words that they do not have any way of understanding.^2
And Ellenberg (2014) on what we call `pinching' knowledge by coming at it from opposing directions:
In fact, it's a common piece of folk advice--I know I heard it from my Ph.D. advisor, and presumably he from his, etc.--that when you're working hard on a theorem you should try to prove it by day and disprove it by night. (The precise frequency of the toggle isn't critical; it's said of the topologist R. H. Bing that his habit was to split each month between two weeks trying to prove the Poincaré Conjecture and two weeks trying to find a counterexample.) Why work at such cross-purposes? There are two good reasons. The first is that you might, after all, be wrong; if the statement you think is true is really false, all your effort to prove it is doomed to be useless. Disproving by night is a kind of hedge against that gigantic waste.
But there's a deeper reason. If something is true and you try to disprove it, you will fail. This is what happened to Bolyai, who bucked his father's well-meaning advice and tried, like so many before him, to prove that the parallel postulate followed from Euclid's other axioms. Like all the others, he failed. But unlike the others, he was able to understand the shape of his failure. What was blocking all his attempts to prove that there was no geometry without the parallel postulate was the existence of just such a geometry! And with each failed attempt he learned more about the features of the thing he didn't think existed, getting to know it more and more intimately, until the moment when he realized it was really there.^3
Finding math and code needs
in evidence (top-down)
AIMA4e:^4
* computer science: problem (complexity) and algorithm (function limiting behavior) analysis (appendix); data structures (p. viii);
* vectors (ordered value sequences), matrices (linear system solving), linear algebra (maps between vector spaces) (appendix);
* probability distributions (appendix);
* BNF; Python, Java (github);
* calculus (p. viii).
Goodfellow et al.:^5
* linear algebra, probability, information theory, numerical methods.
Conference Proceedings: links.retraice.com
ChatGPT, AlphaCode, etc.
Simplified
Our goal is to change state from outsider-non-player to player in the computer control game.^6
But there are many other legitimate reasons to learn AI, as laid out in Retraice (2022/11/27).
Our strategies for learning math and code, and not forgetting our purpose:^7
* Bottom-up: thorough, bookish study of AIMA4e, Deisenroth et al., etc.;
* Top-down: conference and journal papers, public releases;
* Purpose: work on `What's going on out there' by crossing top-down with the hypotheses.^8
The math and code are like conditioning in sports: a matter of early emphasis, and then a lesser but sustained effort throughout the season.
Before the game, needs are more bottom-up, just-in-case; during the game, all needs are top-down, just-in-time.
_
References
Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford. First published in 2014. Citations are from the pbk. edition, 2016. ISBN: 978-0198739838. Searches:
https://www.amazon.com/s?k=978-0198739838
https://www.google.com/search?q=isbn+978-0198739838
https://lccn.loc.gov/2015956648
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge University Press. ISBN: 978-1108455145.
https://mml-book.github.io/ Searches:
https://www.amazon.com/s?k=9781108455145
https://www.google.com/search?q=isbn+9781108455145
https://lccn.loc.gov/2019040762
Ellenberg, J. (2014). How Not to Be Wrong: The Power of Mathematical Thinking. Penguin. ISBN: 978-0143127536. Searches:
https://www.amazon.com/s?k=978-0143127536
https://www.google.com/search?q=isbn+978-0143127536
https://lccn.loc.gov/2014005394
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. ISBN 978-0262035613. Ebook available at:
https://www.deeplearningbook.org/ Searches:
https://www.amazon.com/s?k=978-0262035613
https://www.google.com/search?q=isbn+978-0262035613
https://lccn.loc.gov/2016022992
Retraice (2022/03/07). Re17: Hypotheses to Eleven. retraice.com.
https://www.retraice.com/segments/re17 Retrieved 17th Mar. 2022.
Retraice (2022/10/19). Re22: Computer Control. retraice.com.
https://www.retraice.com/segments/re22 Retrieved 19th Oct. 2022.
Retraice (2022/11/16). Re52: Big Questions About AI. retraice.com.
https://www.retraice.com/segments/re52 Retrieved 17th Nov. 2022.
Retraice (2022/11/17). Re53: Big Questions About Strategic Intelligence. retraice.com.
https://www.retraice.com/segments/re53 Retrieved 18th Nov. 2022.
Retraice (2022/11/18). Re54: Implications and Endgames. retraice.com.
https://www.retraice.com/segments/re54 Retrieved 19th Nov. 2022.
Retraice (2022/11/19). Re55: The Computer Control Game. retraice.com.
https://www.retraice.com/segments/re55 Retrieved 20th Nov. 2022.
Retraice (2022/11/20). Re56: A Valuable Brick: `Artificial Intelligence: A Modern Approach' 4th ed. retraice.com.
https://www.retraice.com/segments/re56 Retrieved 21st Nov. 2022.
Retraice (2022/11/27). Re63: Seventeen Reasons to Learn AI. retraice.com.
https://www.retraice.com/segments/re63 Retrieved Monday Nov. 2022.
Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach. Pearson, 4th ed. ISBN: 978-0134610993. Searches:
https://www.amazon.com/s?k=978-0134610993
https://www.google.com/search?q=isbn+978-0134610993
https://lccn.loc.gov/2019047498
Footnotes
^1 Mentioned during the livestream: Bostrom (2014) p. 23 says 23,000 of 160,000 students completed Thrun and Norvig's 2011 AI online course.
^2 Deisenroth et al. (2020) p. 5.
^3 Ellenberg (2014) pp. 433-434.
^4 Russell & Norvig (2020).
^5 Goodfellow et al. (2016).
^6 Retraice (2022/11/16); Retraice (2022/11/17); Retraice (2022/11/18); Retraice (2022/11/19); Retraice (2022/11/20).
^7 "To forget one's purpose is the commonest form of stupidity." Always attributed to Nietzsche, e.g. in `The Commonest Form of Stupidity', Carlos Reyes, wsj.com Apr. 29, 2011. But I can never find the original quote (I've looked).
^8 Retraice (2022/03/07); Retraice (2022/10/19).
