Training compute (FLOP)
8
equals
9
When C is set to the estimated compute for GPT-3, 3.14E+23, and N is set to GPT-3's number of parameters, 175E+9, then the predicted loss is approximately 2.0025.
10
ML practitioners want to minimize the loss they can achieve on a given compute budget, i.e. achieve the minimum of L(N). If we decrease C until the minimum of L(N) coincides with y = 2.0025 (the loss predicted for GPT-3 by the Hoffman scaling law), the resulting value of C is approximately 1.05E+23 FLOP and the value of N at that minimum is approximately 15E+9 parameters. In turn, the resulting value of D is 1.05E+23 / (6*15E+9) ~= 1.17E+12 tokens.
11
negative 10
10
12
Loss
13
negative 10
10
14
GPT-3 number of parameters
15
equals
16
17
powered by
powered by
New Blank Graph
Examples
Lines: Slope Intercept Form
example
Lines: Point Slope Form
example
Lines: Two Point Form
example
Parabolas: Standard Form
example
Parabolas: Vertex Form
example
Parabolas: Standard Form + Tangent
example
Trigonometry: Period and Amplitude
example
Trigonometry: Phase
example
Trigonometry: Wave Interference
example
Trigonometry: Unit Circle
example
Conic Sections: Circle
example
Conic Sections: Parabola and Focus
example
Conic Sections: Ellipse with Foci
example
Conic Sections: Hyperbola
example
Polar: Rose
example
Polar: Logarithmic Spiral
example
Polar: Limacon
example
Polar: Conic Sections
example
Parametric: Introduction
example
Parametric: Cycloid
example
Transformations: Translating a Function
example
Transformations: Scaling a Function
example
Transformations: Inverse of a Function
example
Statistics: Linear Regression
example
Statistics: Anscombe's Quartet
example
Statistics: 4th Order Polynomial
example
Lists: Family of sin Curves
example
Lists: Curve Stitching
example
Lists: Plotting a List of Points
example
Calculus: Derivatives
example
Calculus: Secant Line
example
Calculus: Tangent Line
example
Calculus: Taylor Expansion of sin(x)
example
Calculus: Integrals
example
Calculus: Integral with adjustable bounds
example
Calculus: Fundamental Theorem of Calculus
example
Terms of Service|Privacy Policy