Suppose you want to find the maximum or minimum of a function but you’re not allowed to explore all of — instead, you’re restricted to points that satisfy a constraint A concrete example: We’re looking for the point(s) (x, y) on the unit circle where is as big or small as possible. Contour Intuition The […]
Lagrange multipliers and optimisation Intuition Read More »
There are two commonly seen definitions of the dot product, which at first glance don’t seem obviously related: A bridge between these two perspectives comes from thinking about linear maps — particularly, the linear map that projects any vector onto a line defined by a unit vector . This framing is nicely illustrated in 3Blue1Brown’s
Dot product intuition Read More »
There are many different ways to interpret the linear regression framework. Below I document the ones I have found useful. The general problem is that we want to predict some variable Y based on some inputs X1 up to Xp. If we assume this relationship is linear then we would have Note here we are
I often get lost in the indices when dealing with differentiation wrt multi-dimensional quantities. So writing up some intuitions and identities that are useful. The cheatsheet from Imperial College Department of Computing is also quite useful (at the bottom of this post): Gradient points in direction of steepest ascent Consider a function or two variables
Matrix calculus cheatsheet Read More »
Given vector spaces V and W, we write L(V, W) := the set of all linear maps from V to W. This is a vector space. Some useful properties for a linear map T in L(V, W): Matrix of a linear map The matrix M of a linear map T from V –> W, w.r.t
Linear maps, bases and matrices Read More »
It is useful to consider vector spaces over C (complex numbers) rather than over R but in general can think of them as over a field F. The motivation fo the definition of a vector spaces comes from the properties of addition and scalar multiplication in Fn. It is tempting to associate the elements of
Finite Dimensional Vector spaces Read More »
Information To let you know whether I am hungry or not I could use the code “0 = not hungry” and “1 = hungry”. This is “1 bit” of information. If a given day has 50% chance of being sunny and 50% chance of rainy. When you learn that it is a sunny day today,
Information, Entropy and Mutual Informations Read More »