Sparse grid methods (continued)

Sparse grid methods (continued) Vesa Kaarnioja (kaarnioja@uni-mannheim.de) Claudia Schillings (c.schillings@uni-mannheim.de) October 14, 2019 University of Mannheim University of Mannheim Sparse grid methods October 14, 2019 1 / 32

Short recap of last week University of Mannheim Sparse grid methods October 14, 2019 2 / 32

For any multi-index α N d (here 0 N), we define α 1 = d α i and α = max 1 i d α i. We consider the function spaces H r (Ω)= { f : Ω R; α f (x) x α } exists and is bounded in Ω for all α r for a fixed region Ω R d. We call r the regularity of functions in H r (Ω) and accompany these function spaces with the respective norms { } α f (x) f H r (Ω) = max sup α N d x α ; x Ω. α r Let T : H r (Ω) R be a bounded, linear functional. The operator norm of T is defined by T := sup f H r (Ω) 1 Tf. Naturally, Tf T f H r (Ω) for all f H r (Ω). University of Mannheim Sparse grid methods October 14, 2019 3 / 32

Suppose that Ω R d 1 and Ξ R d 2 and let S : H r (Ω) R and T : H r (Ξ) R be functionals. Suppose additionally that they admit to representations Sf = m a i f (x i ) and T f = n b i f (y i ) for a selection of positive weights (a i ) m and (b i) n and vectors (x i) m and (y i ) n in the domains Ω and Ξ. Now Ω Ξ Rd 1+d 2 and the tensor product of S and T is the linear functional S T : H r (Ω Ξ) R defined by setting (S T )f = m n a i b j f (x i, y j ). j=1 University of Mannheim Sparse grid methods October 14, 2019 4 / 32

During this lecture, we shall consider the sequence of univariate Clenshaw Curtis quadrature rules (U i ), which are of the form m i U i f = wj i f (xj i ), f H r ([ 1, 1] d ). j=1 The i th CC rule has m i -points, where m 1 = 1 and m i = 2 i 1 + 1, i > 1. The nodes (xj i)m i j=1 and weights (w i j )m i j=1 of an m i-point CC rule have explicit formulae ( ) π(j 1) xj i = cos, j {1,..., m i } m i 1 1 wj i = wm i m i (m i 2) (, j {1, m i} i +1 j = 2 m i 1 cos(π(j 1)) 1 m i (m i 2 ( )) (m i 3)/2 1 2) k=1 cos 2πk(j 1) 4k 2 1 m i otherwise. 1 The evaluation points of the CC rules are nested: Let us denote X i := {x i j }m i j=1. Then X i X i+1 for all i 1. University of Mannheim Sparse grid methods October 14, 2019 5 / 32

Properties of tensor products of linear functionals Noncommutative: generally S T T S. Associative: (S T ) R = S (T R). Distributive: (S + T ) R = S R + T R. Theorem Let T i be quadrature operators. Then n T i = in their respective operator norms. n T i If f (x, y) = g(x)h(y), then (S T )f = Sg Th. University of Mannheim Sparse grid methods October 14, 2019 6 / 32

Definition (Smolyak quadrature rule) Let (U i ) be a sequence of univariate quadrature rules in the interval I R. We introduce the difference operators by setting 0 = 0, 1 = U 1 and i+1 = U i+1 U i for i = 1, 2, 3,.... The Smolyak quadrature rule of order k in the hyperrectangle I d = I I is the operator Q d k = α 1 k α N d d αi. (1) The tensor product α1 αd in the summand of (1) vanishes whenever α i = 0 for some index i. In the sequel we always assume that α 1 and hence k d. University of Mannheim Sparse grid methods October 14, 2019 7 / 32

Proposition (Dimension recursion) Let k d 2. Then Q d k = α 1 k 1 α N d 1, α 1 ( d 1 ) αi U k α 1 = A (computationally) useful characterization: Theorem (Combination method) k 1 i=d 1 Q d 1 i k i. Let U i be univariate quadrature rules in the interval I R and suppose that k d. Then Q d k = max{d,k d+1} α 1 k ( 1) k α 1 ( ) d d 1 U k α αi. (2) 1 University of Mannheim Sparse grid methods October 14, 2019 8 / 32

Exercise. Using the formula Q d k = max{d,k d+1} α 1 k ( 1) k α 1 ( d 1 k α 1 ) d expand the above expression to find the Smolyak quadrature rule Q 2 3. Next, plug the Clenshaw Curtis quadrature rules U 1 f = f ( 1 2 ) and U 2f = 1 6 f (0) + 2 3 f ( 1 2 ) + 1 6 f (1) U αi into the formula you obtained for Q 2 3 form and derive a quadrature rule in the Q 2 3f = w 1 f (x 1 ) + w 2 f (x 2 ) + w 3 f (x 3 ) + w 4 f (x 4 ) + w 5 f (x 5 ), where (w i ) 5 are weights and (x i) 5 are elements in [0, 1]2. University of Mannheim Sparse grid methods October 14, 2019 9 / 32

Solution. You should obtain (akin to last week s example) Q 2 3 = U 1 U 2 + U 2 U 1 U 1 U 1. Substituting the Clenshaw Curtis rules yields Q 2 3f = 1 6 f ( 1 2, 0) + 1 6 f ( 1 2, 1) + 1 6 f (0, 1 2 ) + 1 6 f (1, 1 2 ) + 1 3 f ( 1 2, 1 2 ). University of Mannheim Sparse grid methods October 14, 2019 10 / 32

Today, we consider the problem of approximating I d f := f (y 1,..., y d ) dy 1 dy d, f H r ([ 1, 1] d ). [ 1,1] d To this end, let us extend the definition of the tensor product as follows: if Tf = n w if (x i ) for f H r ([ 1, 1] s ), then we define (T I d )f = (I d T )f = n w i f (x i, y) dy [ 1,1] s n w i f (y, x i ) dy [ 1,1] s (I d I s )f = f (x, y) dx dy [ 1,1] s+d for f H r ([ 1, 1] d+s ). Note that I d T = T I d = I d T. University of Mannheim Sparse grid methods October 14, 2019 11 / 32

Some essential combinatorics Lemma α 1 =k 1 = ( ) k 1 d 1 and α 1 k 1 = ( ) k. d Proof. The first identity follows from a combinatorial ( stars and bars ) argument. The second identity follows from α 1 k 1 = k l=d α 1 =l 1 = k l=d ( ) l 1 = d 1 ( ) k, d where the final equality follows from the summation formula for the diagonals of Pascal s triangle (proof by induction and using Pascal s identity; omitted.) University of Mannheim Sparse grid methods October 14, 2019 12 / 32

On the distribution of Smolyak quadrature nodes The evaluation points of Q d k form the set η(k, d) = X α1 X αd for all k d. max{d,k d+1} α 1 k The elements of the set η(k, d) are the nodes of Q d k. If the univariate rules are nested, i.e., X i X i+1 (as is the case with CC rules), then the nodes of Q d k form the set η(k, d) = X α1 X αd for all k d. (3) α 1 =k University of Mannheim Sparse grid methods October 14, 2019 13 / 32

1 0.8 0.6 0.4 0.2 0-0.2-0.4-0.6-0.8-1 -1-0.8-0.6-0.4-0.2 0 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0-0.2-0.4-0.6-0.8-1 -1-0.8-0.6-0.4-0.2 0 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0-0.2-0.4-0.6-0.8-1 -1-0.8-0.6-0.4-0.2 0 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0-0.2-0.4-0.6-0.8-1 -1-0.8-0.6-0.4-0.2 0 0.2 0.4 0.6 0.8 1 Figure: Left: non-nested Smolyak Gauss Legendre rules Q d k for d = 2 and k {5, 6}. Right: nested Smolyak Gauss Patterson rules Q d k for d = 2 and k {5, 6}. University of Mannheim Sparse grid methods October 14, 2019 14 / 32

Error analysis University of Mannheim Sparse grid methods October 14, 2019 15 / 32

Note that I d f Q d k f Id Q d k f H r (Ω). We can estimate the worst case error of the Smolyak quadrature rule Q d k by bounding the operator norm of I d Q d k. University of Mannheim Sparse grid methods October 14, 2019 16 / 32

It is known from classical approximation theory that the sequence of univariate CC rules satisfy for some sequence of numbers (γ r ) r 0. I 1 U k γ r 2 rk (4) Lemma The Smolyak Clenshaw Curtis quadrature rules satisfy the bound ( ) k I d Q d k γ r max{2 r+1, γ r (1 + 2 r )} d 1 2 rk. d 1 Proof. By dimension-wise induction. The case d = 1 is immediate since (4) and the telescoping property imply that ( ) k I 1 Q 1 k = I1 U k γ r 2 rk γ r max{2 r+1, γ r (1 + 2 r )} }{{} 1 1 1 }{{} =1 as desired. 2 rk University of Mannheim Sparse grid methods October 14, 2019 17 / 32

Next assume that the claim ( ) I d Q d k γ r max{2 r+1, γ r (1 + 2 r )} d 1 k d 1 holds for some d 1. Then I d+1 Q d+1 k+1 = I d I 1 Q d k I 1 + Q d k I 1 Q d+1 k+1 ( d ) = (I d Q d k ) I 1 + αi I 1 = (I d Q d k ) I 1 + α 1 k α 1 k Taking the operator norm α 1 k ( d ) αi (I 1 U k+1 α 1 ). I d+1 Q d+1 k+1 I d Q d k I 1 + α 1 k ( d 2 rk ( d ) αi U k+1 α 1 ) αi I 1 U k+1 α 1. University of Mannheim Sparse grid methods October 14, 2019 18 / 32

Here we have I 1 = 2, I 1 U k+1 α 1 γ r 2 r(k+1 α 1), and I d Q d k γ r max{2 r+1, γ r (1 + 2 r )} d 1( k d 1) 2 rk by the induction assumption. Noting that αi U αi I 1 + I 1 U αi 1 γ r 2 rα i + γ r 2 rα i +r = γ r 2 rα i (1 + 2 r ) we obtain ( d ) αi α 1 k }{{} γr d 2 r α 1 (1+2 r ) d I 1 U k+1 α 1 }{{} γ r 2 r(k+1 α 1 ) = α 1 k γr d+1 2 r(k+1) (1 + 2 r ) d ( ) k γr d+1 2 r(k+1) (1 + 2 r ) d. d University of Mannheim Sparse grid methods October 14, 2019 19 / 32

Hence I d+1 Q d+1 k+1 I d Q d k I 1 + α 1 k ( d ) αi I 1 U k+1 α 1 ( ( ) 2γ r max{2 r+1, γ r (1 + 2 r )} d 1 k )2 rk + γ r [γ r (1 + 2 r )] d k 2 r(k+1) d 1 d ( ( ) 2 r+1 γ r max{2 r+1, γ r (1 + 2 r )} d 1 k )2 r(k+1) + γ r [γ r (1 + 2 r )] d k 2 r(k+1) d 1 d ( ) [ ( ) ( ) ] γ r max{2 r+1, γ r (1 + 2 r )} d 2 r(k+1) k k + (Pascal s identity) d 1 d ( ) = γ r max{2 r+1, γ r (1 + 2 r )} d 2 r(k+1) k + 1 d proving the assertion. ( ) Note that here x max{x, y} d 1 max{x, y} d for x, y 0. University of Mannheim Sparse grid methods October 14, 2019 20 / 32

A corollary to the previous Lemma is Smolyak s original estimate I d Q d k Ckd 1 2 rk, where the constant C > 0 depends on r and d. One can relate this error bound (depending on the level k) to the number of evaluation points N = N(k, d) of the Smolyak Clenshaw Curtis rule Q d k. Recalling that the CC rules are nested, we get (cf. (3)) N m α1 m αd 2 α 1 = 2 k 1 α 1 =k α 1 =k = 2 k ( k 1 d 1 α 1 =k ) C 2 k k d 1, where C > 0 is a constant that depends on d and we used m i 2 i as well as the Lemma on slide 9. University of Mannheim Sparse grid methods October 14, 2019 21 / 32

For simplicity, let us write a b if a Cb for some constant C > 0. Since N 2 k k d 1, we have 2 k kd 1 N. Hence, the Smolyak error term can be recast as I d Q d k 2 rk k d 1 kr(d 1) N r k d 1 = k(r+1)(d 1) N r. For sufficiently large k, we have 2 k N and hence k log N; therefore I d Q d (log N)(r+1)(d 1) k N r, where the implied coefficient depends on r and d. Hence, for fixed regularity r and fixed d, the method converges at the above rate as N. University of Mannheim Sparse grid methods October 14, 2019 22 / 32

Polynomial exactness University of Mannheim Sparse grid methods October 14, 2019 23 / 32

P d k = { R d x β 1 k β N d { d P 1 m i = R d (x 1,..., x d ) Lemma a β x β R; a β R for all β N d }. } d p i (x i ) R; p i P 1 m i for i = 1,..., d. The Smolyak Clenshaw Curtis rules satisfy Q d k f = Id f for all polynomials f α 1 =k α N d P mα1 P mαd. Proof. By dimension-wise induction. For d = 1, the claim is reduced into I 1 f = U k f for all f Π k. This is true since the CC rule is interpolatory. University of Mannheim Sparse grid methods October 14, 2019 24 / 32

Suppose that the claim is true for some d 1. Let β N d+1 such that β 1 = k and k d + 1. Define f (x 1,..., x d+1 ) = g(x 1,..., x d )f d+1 (x d+1 ), where g(x 1,..., x d ) = f 1 (x 1 ) f d (x d ) and f i P 1 m i for i = 1,..., d + 1. Now clearly f d+1 P1 m βi. It is sufficient to prove the claim for the function f since linearity of the Smolyak rule implies that the claim then holds for any element in d+1 α 1 =k P1 m αi as well. α N d+1 Using dimension recursion and the product structure of f we get k 1 Q d+1 k f = Q d i k i f = i=d k 1 Q d i g k i f d+1. If β d+1 k i 1, then m βd+1 m k i 1 m k i and we have U k i f d+1 = U k i 1 f d+1 = I 1 f d+1. Especially k i f d+1 = 0 and we can truncate the expression for Q d+1 k by considering summation over the indices k β d+1 i k 1. University of Mannheim Sparse grid methods October 14, 2019 25 / 32 i=d

Using the fact that k = β 1 allows us to write the rule Q d+1 k in the form Q d+1 k f = k 1 i=β 1 + +β d Q d i g k i f d+1. Our induction hypothesis implies that I d g = Q d i g for β 1 + + β d i k 1 and we achieve proving the claim. Q d+1 k f = k 1 i=β 1 +...+β d I d g k i f d+1 = I d g U k β1 β d f d+1 = I d g U βd+1 f d+1 = I d g I 1 f d+1 = I d+1 f University of Mannheim Sparse grid methods October 14, 2019 26 / 32

Relating the polynomial exactness to the total degree m of classical polynomial spaces P d m is rather technical the explicit expression for the total degree m of exactness in terms of the dimension d and level k of a Smolyak Clenshaw Curtis rule Q d k is somewhat complicated, and can be found in [Novak & Ritter (1999)]. The following (weaker) assertion, however, is true for Smolyak Clenshaw Curtis rules. Theorem (Corollary 1 in [Novak & Ritter (1999)]) The Smolyak Clenshaw Curtis rule has (at least) a degree 2(k d) + 1 of exactness. University of Mannheim Sparse grid methods October 14, 2019 27 / 32

Discussion University of Mannheim Sparse grid methods October 14, 2019 28 / 32

During these past two lectures, we have considered the construction and approximation properties of classical, isotropic Smolyak quadrature rules We have seen that Q d k = α 1 k α N d d αi. there exists a computationally useful combination method that can be used to effectively implement the isotropic Smolyak rule. the Smolyak Clenshaw Curtis rule has an error rate ( ) (log N) (d 1)(r+1) O for functions f H r ([ 1, 1] s ). the Smolyak Clenshaw Curtis rule is exact for polynomials of total degree (at least) 2(k d) + 1. N r University of Mannheim Sparse grid methods October 14, 2019 29 / 32

Extensions In practice, many of these approximation results extend to other, more general quadrature rules Ũ (j) k (j) mk f = w i f (x i ) W j (x)f (x) dx, I j (5) with different pairings of intervals I j R and weight functions W j (x) 0 (with finite moments). In this case, the Smolyak construction approximates integrals of the form I 1 I d W 1 (x 1 ) W d (x d )f (x 1,..., x d ) dx 1 dx d. Similar results on convergence and polynomial approximation can be obtained as long as the univariate rules (5) are interpolatory and the sequence (m (j) i ) is imposed a sufficient growth condition. It is generally more economical to use nested univariate formulae. University of Mannheim Sparse grid methods October 14, 2019 30 / 32

Other extensions Depending on the application, one may be interested to consider generalized sparse grid constructions d (A αi A αi 1). α I The operators (A i ) can be replaced, e.g., by interpolation operators or projection operators. Note that the convergence rates may differ from the quadrature setting! In many situations, the components of your function may have different, relative importance or anisotropy. For example, x 1 affects the integration problem more than x 2, x 2 affects the result more than x 3, etc. The index set I can therefore be either tailored to fit the a priori information of your problem, or one can use a dimensionadaptive scheme. The combination method (2) no longer works! These approaches will be featured in the upcoming talks! University of Mannheim Sparse grid methods October 14, 2019 31 / 32

References The combination method and fundamental error analysis: G. Wasilkowski and H. Woźniakowski. Explicit cost bounds of algorithms for multivariate tensor product problems. Journal of Complexity 11:1 56, 1995. On the approximation properties of Smolyak Clenshaw Curtis rules: E. Novak and K. Ritter. High dimensional integration of smooth functions over cubes. Numerische Mathematik 75(1):79 97, 1996. E. Novak and K. Ritter. Simple cubature formulas with high polynomial exactness. Constructive Approximation 15(4):499 522, 1999. Additional reading: M. Holtz. Sparse Grid Quadrature in High Dimensions with Applications in Finance and Insurance. Springer, Heidelberg, 2011. University of Mannheim Sparse grid methods October 14, 2019 32 / 32