{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "89892f6e", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "using FundamentalsNumericalComputation\n", "FNC.init_format()\n", "disable_logging(Logging.Warn)" ] }, { "cell_type": "markdown", "id": "5a872c2c", "metadata": {}, "source": [ "(section-linsys-efficiency)=\n", "# Efficiency of matrix computations\n", "\n", "Predicting how long an algorithm will take to solve a particular problem, on a particular computer, as written in a particular way in a particular programming language, is an enormously difficult undertaking. It's more practical to predict how the required time will scale as a function of the size of the problem. In the case of a linear system of equations, the problem size is $n$, the number of equations and variables in the system. Because expressions of computational time are necessarily approximate, it's customary to suppress all but the term that is dominant as $n\\to\\infty$. We first need to build some terminology for these expressions.\n", "\n", "{index} ! asymptotic notation\n", "\n", "\n", "## Asymptotic analysis\n", "\n", "::::{proof:definition} Asymptotic notation\n", "Let $f(n)$ and $g(n)$ be positive-valued functions. We say $f(n)=O(g(n))$ (read \"$f$ is **big-O** of $g$\") as $n\\rightarrow \\infty$ if $f(n)/g(n)$ is bounded above as $n\\to\\infty$.\n", "\n", "We say $f(n)\\sim g(n)$ (read \"$f$ is **asymptotic** to $g$\") as $n\\rightarrow \\infty$ if $f(n)/g(n)\\rightarrow 1$ as $n\\rightarrow\\infty$.\n", "::::\n", "\n", "One immediate consequence is that $f\\sim g$ implies $f=O(g)$.[^sets]\n", "\n", "[^sets]: More precisely, $O(g)$ and $\\sim g$ are *sets* of functions, and $\\sim g$ is a subset of $O(g)$. That we write $f=O(g)$ rather than $f\\in O(g)$ is a quirk of convention.\n", "\n", "{proof:example}\n", "Consider the functions $f(n) = a_1 n^3 + b_1 n^2 + c_1 n$ and $g(n) = a_2 n^3$ in the limit $n\\to \\infty$. Then\n", " \n", "{math}\n", " \\lim_{n \\to \\infty} \\frac{f(n)}{g(n)}\n", " = \\lim_{n \\to \\infty} \\frac{a_1 + b_1n^{-1} + c_1n^{-2}}{a_2} =\n", " \\frac{a_1}{a_2} .\n", "\n", "\n", "Since $a_1/a_2$ is a constant, $f(n) = O(g(n))$; if $a_1=a_2$, then $f \\sim g$.\n", "\n", "\n", "{proof:example}\n", " Consider $f(n) = \\sin (1/n)$, $g(n)=1/n$ and $h(n) = 1/n^2$. For large $n$, Taylor's theorem with remainder implies that\n", " \n", "{math}\n", "f(n) = \\frac{1}{n} - \\cos(1/\\xi)\\frac{1}{6 n^3},\n", "\n", "\n", "where $n<\\xi<\\infty$. But\n", " \n", "{math}\n", "\\lim_{n\\to \\infty} \\frac{f}{g} = \\lim_{n\\to \\infty} 1-\\cos(1/\\xi)\\frac{1}{6 n^2} = 1,\n", "\n", "\n", "and so $f \\sim g$. On the other hand, comparing $f$ and $h$, we find\n", "\n", "{math}\n", "\\lim_{n\\to \\infty} \\frac{f}{h} = \\lim_{n\\to \\infty} n-\\cos(1/\\xi)\\frac{1}{6 n} = \\infty,\n", "\n", "\n", "so we cannot say that $f = O(h)$. A consideration of $h/f$ will show that $h = O(f)$, however.\n", "\n", "\n", "It's conventional to use asymptotic notation that is as specific as possible. For instance, while it is true that $n^2+n=O(n^{10})$, it's more informative, and usually expected, to say $n^2+n=O(n^2)$. There are additional notations that enforce this requirement strictly, but we will just stick to the informal understanding.\n", "\n", "There is a memorable way to use asymptotic notation to simplify sums:\n", "\n", "{math}\n", ":label: sumflops\n", "\\begin{split}\n", " \\sum_{k=1}^n k&\\sim \\frac{n^2}{2} = O(n^2), \\text{ as $n\\to\\infty$}, \\\\\n", " \\sum_{k=1}^n k^2 &\\sim \\frac{n^3}{3} = O(n^3), \\text{ as $n\\to\\infty$}, \\\\\n", " &\\vdots \\\\\n", " \\sum_{k=1}^n k^p &\\sim \\frac{n^{p+1}}{p+1} = O(n^{p+1}), \\text{ as $n\\to\\infty$}.\n", "\\end{split}\n", "\n", "\n", "These formulas greatly resemble the definite integral of $x^p$. \n", "\n", "(example-efficiency-sums)=\n", "::::{proof:example}\n", "\\begin{align*}\n", "\\sum_{k=1}^{n-1} 4k^2 + 3 & = 4 \\left( \\sum_{k=1}^{n-1} k^2\\right) + 3 \\sum_{k=1}^{n-1} 1\\\\ \n", "&\\sim 4 \\left( \\frac{1}{3} (n-1)^3 \\right) + 3(n-1) \\\\ \n", "& = \\frac{4}{3} (n^3 - 3n^2 + 3n - 1) + 3n - 3 \\\\ \n", "&\\sim \\frac{4}{3} n^3.\n", "\\end{align*}\n", "::::\n", "\n", "## Flop counting\n", "\n", "{index} flops\n", "\n", "\n", "Traditionally, in numerical linear algebra we count **floating-point operations**, or **flops** for short. In our interpretation each scalar addition, subtraction, multiplication, division, and square root counts as one flop. Given any algorithm, we simply add up the number of scalar flops and ignore everything else.\n", "\n", "(demo-flops-mvmult)=\n", "{proof:demo}\n", "\n", "\n", "{raw} html\n", "
\n", "\n", "\n", "{raw} latex\n", "%%start demo%%\n", "\n", "\n", "\n", "Here is a straightforward implementation of matrix-vector multiplication." ] }, { "cell_type": "code", "execution_count": null, "id": "3dc1c88e", "metadata": {}, "outputs": [], "source": [ "n = 6\n", "A,x = randn(n,n),rand(n)\n", "y = zeros(n)\n", "for i in 1:n\n", " for j in 1:n\n", " y[i] += A[i,j]*x[j] # 1 multiply, 1 add\n", " end\n", "end" ] }, { "cell_type": "markdown", "id": "96d87e73", "metadata": {}, "source": [ "Each of the loops implies a summation of flops. The total flop count for this algorithm is\n", "\n", "$$\\sum_{i=1}^n \\sum_{j=1}^n 2 = \\sum_{i=1}^n 2n = 2n^2.$$\n", "\n", "Since the matrix $\\mathbf{A}$ has $n^2$ elements, all of which have to be involved in the product, it seems unlikely that we could get a flop count that is smaller than $O(n^2)$ in the general case.\n", "\n", "{index} ! Julia; push\\!, ! Julia; for\n", "\n", "\n", "::::{panels}\n", ":column: col-7 left-side\n", ":card: border-0 shadow-none\n", "{raw} latex\n", "\\begin{minipage}[t]{0.5\\textwidth}\n", "\n", "Let's run an experiment with the built-in matrix-vector multiplication. Note that Julia is unusual in that loops have a variable scope separate from its enclosing code. Thus, for n in n below means that inside the loop, the name n will take on each one of the values that were previously assigned to the vector n.\n", "\n", "{raw} latex\n", "\\end{minipage}\\hfill\n", "\n", "---\n", ":column: col-5 right-side\n", ":card: shadow-none comment\n", "{raw} latex\n", "\\begin{minipage}[t]{0.4\\textwidth}\\begin{mdframed}[default]\\small\n", "\n", "The push! function attaches a new value to the end of a vector.\n", "{raw} latex\n", "\\end{mdframed}\\end{minipage}\n", "\n", "::::" ] }, { "cell_type": "code", "execution_count": null, "id": "a030bebc", "metadata": {}, "outputs": [], "source": [ "n = 1000:1000:5000\n", "t = []\n", "for n in n\n", " A = randn(n,n) \n", " x = randn(n)\n", " time = @elapsed for j in 1:80; A*x; end\n", " push!(t,time)\n", "end" ] }, { "cell_type": "markdown", "id": "855a709f", "metadata": {}, "source": [ "The reason for doing multiple repetitions at each value of $n$ in the loop above is to avoid having times so short that the resolution of the timer is significant." ] }, { "cell_type": "code", "execution_count": null, "id": "e78c5efe", "metadata": {}, "outputs": [], "source": [ "pretty_table((n=n,t=t),[\"size\" \"time\";\"\" \"(sec)\"])" ] }, { "cell_type": "markdown", "id": "f3cf4efd", "metadata": {}, "source": [ "{index} Julia; Boolean indexing\n", "\n", "\n", "::::{panels}\n", ":column: col-7 left-side\n", ":card: border-0 shadow-none\n", "{raw} latex\n", "\\begin{minipage}[t]{0.5\\textwidth}\n", "\n", "Looking at the timings just for $n=2000$ and $n=4000$, they have ratio\n", "\n", "{raw} latex\n", "\\end{minipage}\\hfill\n", "\n", "---\n", ":column: col-5 right-side\n", ":card: shadow-none comment\n", "{raw} latex\n", "\\begin{minipage}[t]{0.4\\textwidth}\\begin{mdframed}[default]\\small\n", "\n", "The expression n.==4000 here produces a vector of Boolean (true/false) values the same size as n. This result is used to index within t, accessing only the value for which the comparison is true.\n", "{raw} latex\n", "\\end{mdframed}\\end{minipage}\n", "\n", "::::" ] }, { "cell_type": "code", "execution_count": null, "id": "aaa9b7b2", "metadata": {}, "outputs": [], "source": [ "@show t[n.==4000] ./ t[n.==2000];" ] }, { "cell_type": "markdown", "id": "d0b79eb1", "metadata": {}, "source": [ "If the run time is dominated by flops, then we expect this ratio to be $(4000)^2/(2000)^2=4$.\n", "{raw} html\n", "
\n", "\n", "\n", "{raw} latex\n", "%%end demo%%\n", "\n", "\n", "Suppose that the running time $t$ of an algorithm obeys a function that is $O(n^p)$. For sufficiently large $n$, $t\\approx Cn^p$ for a constant $C$ should be a good approximation. Hence\n", "\n", "{math}\n", " :label: loglogfit\n", " t \\approx Cn^p \\qquad \\Longleftrightarrow \\qquad \\log t \\approx p(\\log n) + \\log C.\n", "\n", "\n", "So we expect that a graph of $\\log t$ as a function of $\\log n$ will be a straight line of slope $p$.\n", "\n", "\n", "(demo-flops-loglog)=\n", "{proof:demo}\n", "\n", "\n", "{raw} html\n", "
\n", "\n", "\n", "{raw} latex\n", "%%start demo%%\n", "\n", "\n", "Let's repeat the experiment of the previous figure for more, and larger, values of $n$." ] }, { "cell_type": "code", "execution_count": null, "id": "f7a564ce", "metadata": {}, "outputs": [], "source": [ "randn(5,5)*randn(5); # throwaway to force compilation\n", "\n", "n = 400:200:6000\n", "t = []\n", "for n in n\n", " A = randn(n,n) \n", " x = randn(n)\n", " time = @elapsed for j in 1:50; A*x; end\n", " push!(t,time)\n", "end" ] }, { "cell_type": "markdown", "id": "04c3b6ff", "metadata": {}, "source": [ "Plotting the time as a function of $n$ on log-log scales is equivalent to plotting the logs of the variables." ] }, { "cell_type": "code", "execution_count": null, "id": "cbeb4226", "metadata": {}, "outputs": [], "source": [ "scatter(n,t,label=\"data\",legend=false,\n", " xaxis=(:log10,L\"n\"), yaxis=(:log10,\"elapsed time (sec)\"),\n", " title=\"Timing of matrix-vector multiplications\")" ] }, { "cell_type": "markdown", "id": "75b5a2a9", "metadata": {}, "source": [ "You can see that while the full story is complicated, the graph is trending to a straight line of positive slope. For comparison, we can plot a line that represents $O(n^2)$ growth exactly. (All such lines have slope equal to 2.)" ] }, { "cell_type": "code", "execution_count": null, "id": "38ebae3c", "metadata": {}, "outputs": [], "source": [ "plot!(n,t[end]*(n/n[end]).^2,l=:dash,\n", " label=L\"O(n^2)\",legend=:topleft)" ] }, { "cell_type": "markdown", "id": "65a437ce", "metadata": {}, "source": [ "{raw} html\n", "
\n", "\n", "\n", "{raw} latex\n", "%%end demo%%\n", "\n", "\n", "## Solution of linear systems\n", "\n", "Recall the steps of {numref}Algorithm {number}  for the system $\\mathbf{A}\\mathbf{x}=\\mathbf{b}$:\n", "\n", "1. Factor $\\mathbf{L}\\mathbf{U}=\\mathbf{A}$ using Gaussian elimination.\n", "2. Solve $\\mathbf{L}\\mathbf{z}=\\mathbf{b}$ for $\\mathbf{z}$ using forward substitution.\n", "3. Solve $\\mathbf{U}\\mathbf{x}=\\mathbf{z}$ for $\\mathbf{x}$ using backward substitution.\n", "\n", "The second and third steps are solved by {numref}Function {number}  and {numref}Function {number} . Only one line in each of these functions dominates the arithmetic. Take forwardsub, for instance. It has a single flop in line 11. Line 13 computes\n", "\n", "julia \n", "sum( L[i,j]*x[j] for j in 1:i-1 )\n", "\n", "\n", "This line requires $i-1$ multiplications and $(i-2)$ additions, for a total of $2i-3$ flops. Line 14 adds two more flops. These lines are performed within a loop as $i$ ranges from 1 to $n$, so the total count is\n", "\n", "{math}\n", " :label: trisolveflops\n", " 1 + \\sum_{i=1}^n (2i-3) = 1 - 3n + 2 \\sum_{i=1}^n i.\n", "\n", "\n", "It is not hard to find an exact formula for the sum, but we use {eq}sumflops to simplify it to $\\sim n^2$. After all, since flop counting is only an approximation of true running time, why bother with the more complicated exact expression? An analysis of backward substitution yields the same result.\n", "\n", "{proof:lemma}\n", "Solving a triangular $n\\times n$ system by forward or backward substitution takes $\\sim n^2$ flops asymptotically.\n", "\n", "\n", "Before counting flops for the LU factorization, we have to admit that {numref}Function {number}  is not written as economically as it could be. Recall from our motivating example in {numref}Demo {number}  that we zero out the first row and column of $\\mathbf{A}$ with the first outer product, the second row and column with the second outer product, and so on. There is no good reason to do multiplications and additions with values known to be zero, so we could replace lines 15–19 of lufact with\n", "\n", "{code-block} julia\n", ":lineno-start: 15\n", "for k in 1:n-1\n", " U[k,k:n] = Aₖ[k,k:n]\n", " L[k:n,k] = Aₖ[k:n,k]/U[k,k]\n", " Aₖ[k:n,k:n] -= L[k:n,k]*U[k,k:n]'\n", "end\n", "\n", "\n", "We will use the following handy fact.\n", ":::{proof:observation}\n", "The range k:n, where $k\\le n$, has $n-k+1$ elements.\n", ":::\n", "\n", "Line 17 above divides each element of the vector Aₖ[k:n,k] by a scalar. Hence the number of flops equals the length of the vector, which is $n-k+1$. \n", "\n", "Line 18 has an outer product followed by a matrix subtraction. The definition {eq}def-outerprod of the outer product makes it clear that that computation takes one flop (multiplication) per element of the result, which here results in $(n-k+1)^2$ flops. The number of subtractions is identical. \n", "\n", "Altogether the factorization takes\n", "\n", "{math}\n", ":label: gecount1\n", "\\sum_{k=1}^{n-1} n-k + 1 + 2(n-k+1)^2.\n", "\n", "\n", "There are different ways to simplify this expression. We will make a change of summation index using $j=n-k$. The endpoints of the sum are $j=n-1$ when $k=1$ and $j=1$ when $k=n-1$. Since the order of terms in a sum doesn't matter, we get\n", "\n", "{math}\n", "\\sum_{j=1}^{n-1} 1+j+2(j+1)^2 &= \\sum_{j=1}^{n-1} 3 + 5j + 2j^2 \\\\\n", " & \\sim 3(n-1) + \\frac{5}{2}(n-1)^2 + \\frac{2}{3}(n-1)^3 \\\\\n", " & \\sim \\frac{2}{3}n^3.\n", "\n", "\n", "We have proved the following. \n", "{proof:theorem} Efficiency of LU factorization\n", "The LU factorization of an $n\\times n$ matrix takes $\\sim\\frac{2}{3}n^3$ flops as $n\\to \\infty$. This dominates the flops for solving an $n\\times n$ linear system.\n", "\n", "\n", "(demo-flops-lufact)=\n", "{proof:demo}\n", "\n", "\n", "{raw} html\n", "
\n", "\n", "\n", "{raw} latex\n", "%%start demo%%\n", "\n", "\n", "::::{panels}\n", ":column: col-7 left-side\n", ":card: border-0 shadow-none\n", "{raw} latex\n", "\\begin{minipage}[t]{0.5\\textwidth}\n", "\n", "We'll test the conclusion of $O(n^3)$ flops experimentally, using the built-in lu function instead of the purely instructive lufact.\n", "\n", "{raw} latex\n", "\\end{minipage}\\hfill\n", "\n", "---\n", ":column: col-5 right-side\n", ":card: shadow-none comment\n", "{raw} latex\n", "\\begin{minipage}[t]{0.4\\textwidth}\\begin{mdframed}[default]\\small\n", "\n", "The first time a function is invoked, there may be significant time needed to compile it in memory. Thus, when timing a function, run it at least once before beginning the timing.\n", "{raw} latex\n", "\\end{mdframed}\\end{minipage}\n", "\n", "::::" ] }, { "cell_type": "code", "execution_count": null, "id": "fa7ccbea", "metadata": {}, "outputs": [], "source": [ "lu(randn(3,3)); # throwaway to force compilation\n", "\n", "n = 400:400:4000\n", "t = []\n", "for n in n\n", " A = randn(n,n) \n", " time = @elapsed for j in 1:12; lu(A); end\n", " push!(t,time)\n", "end" ] }, { "cell_type": "markdown", "id": "9389638b", "metadata": {}, "source": [ "We plot the timings on a log-log graph and compare it to $O(n^3)$. The result could vary significantly from machine to machine, but in theory the data should start to parallel the line as $n\\to\\infty$." ] }, { "cell_type": "code", "execution_count": null, "id": "8861611f", "metadata": {}, "outputs": [], "source": [ "scatter(n,t,label=\"data\",legend=:topleft,\n", " xaxis=(:log10,L\"n\"), yaxis=(:log10,\"elapsed time\"))\n", "plot!(n,t[end]*(n/n[end]).^3,l=:dash,label=L\"O(n^3)\")" ] }, { "cell_type": "markdown", "id": "9e9852aa", "metadata": {}, "source": [ "{raw} html\n", "
\n", "\n", "\n", "{raw} latex\n", "%%end demo%%\n", "\n", "\n", "\n", "In practice, flops are not the only aspect of an implementation that occupies significant time. Our position is that counting flops as a measure of performance is a useful oversimplification. We will assume that LU factorization (and as a result, the solution of a linear system of $n$ equations) requires a real-world time that is roughly $O(n^3)$. This growth rate is a great deal more tolerable than, say, $O(2^n)$, but it does mean that for (at this writing) $n$ greater than 10,000 or so, something other than general LU factorization will have to be used.\n", "\n", "## Exercises\n", "\n", "1. ✍ The following are asymptotic assertions about the limit $n\\rightarrow\\infty$. In each case, prove the statement true or false.\n", "\n", " **(a)** $n^2 = O(\\log n),\\quad$ \n", " **(b)** $n^{a} = O(n^b)$ if $a\\le b,\\quad$\n", " **(c)** $e^n \\sim e^{2n},\\quad$\n", " **(d)** $n+\\sqrt{n}\\sim n+2\\sqrt{n}$.\n", "\n", "2. ✍ The following are asymptotic assertions about the limit $h\\to 0$. In each case, prove the statement true or false.\n", "\n", " **(a)** $h^2\\log(h) = O(h^3),\\quad$\n", " **(b)** $h^{a} = O(h^b)$ if $a < b,\\quad$\n", " **(c)** $\\sin(h) \\sim h,\\quad$\n", " **(d)** $(e^{2h}-1)\\sim h$.\n", "\n", "3. ✍ Show that the inner product of two $n$-vectors takes exactly $2n-1$ flops.\n", "\n", "4. ✍ Show that the multiplication of two $n\\times n$ matrices takes $\\sim 2n^3$ flops.\n", "\n", "5. ✍ This problem is about evaluation of a polynomial $c_1 + c_2 x + \\cdots + c_{n}x^{n-1}$.\n", " \n", " **(a)** Here is a little code to do the evaluation.\n", "\n", "  julia\n", " y = c\n", " xpow = 1\n", " for i in 2:n\n", " xpow *= x\n", " y += c[i]*xpow\n", " end\n", " \n", "\n", " Assuming that x is a scalar, how many flops does this function take, as a function of $n$?\n", "\n", " **(b)** Compare the count from (a) to the flop count for Horner's algorithm, {numref}Function {number} .\n", " \n", "6. The exact sums for $p=1,2$ in {eq}sumflops are as follows:\n", " \n", " {math}\n", " \\sum_{k=1}^{n} k = \\frac{n(n+1)}{2}, \\qquad \n", " \\sum_{k=1}^{n} k^2 = \\frac{n(n+1)(2n+1)}{6}.\n", " \n", "\n", " **(a)** ✍ Use these to find the exact result for {eq}gecount1.\n", "\n", " **(b)** ⌨ Plot the ratio of your result from (a) and the asymptotic result $2n^3/3$ for all $n=10^{1+0.03i}$, $i=0,\\dots,100$, using a log scale for $n$ and a linear scale for the ratio. (The curve should approach 1 asymptotically.)\n", " \n", "\n", "7. ✍ Show that for any nonnegative constant integer $m$,\n", " \n", " {math}\n", " \\sum_{k=0}^{n-m} k^p \\sim \\frac{n^{p+1}}{p+1}.\n", " \n", "\n", "8. ⌨ The UpperTriangular and LowerTriangular matrix types cause specialized algorithms to be invoked by the backslash. Define\n", "\n", " \n", " A = rand(1000,1000)\n", " B = tril(A)\n", " C = LowerTriangular(B)\n", " b = rand(1000)\n", " \n", "\n", " Using @elapsed with the backslash solver, time how long it takes to solve the linear system $\\mathbf{A}\\mathbf{x}=\\mathbf{b}$ 100 times; then, do the same for matrices $\\mathbf{B}$ and $\\mathbf{C}$. Is the timing for $\\mathbf{B}$ closer to $\\mathbf{A}$ or to $\\mathbf{C}$? (Hint: Remember to make one timing run without recording results, so that compilation time is not counted.)" ] } ], "metadata": { "jupytext": { "cell_metadata_filter": "-all", "formats": "md:myst", "text_representation": { "extension": ".md", "format_name": "myst", "format_version": 0.13, "jupytext_version": "1.10.3" } }, "kernelspec": { "display_name": "Julia 1.7.1", "language": "julia", "name": "julia-fast" }, "source_map": [ 15, 20, 124, 133, 167, 176, 180, 182, 210, 212, 247, 258, 262, 266, 270, 273, 384, 394, 398, 402 ] }, "nbformat": 4, "nbformat_minor": 5 }