Divide & Conquer

1 Introduction (top-down merge_sort)

Here is the merge_sort algorithm from the Sorts notes (Ch 13 Slide 13):

void merge_sort(A,p,r)
{
  if (r - p > 1)
  {
    q = (p+r)/2;          // n = r-p = size of input
    merge_sort(A,p,q);    // q-p = n/2 = size of subrange
    merge_sort(A,q,r);    // r-q = n/2 = size of subrange
    merge(A,p,q,r);       // Θ(n)
  }
}
void merge(T* A, size_t p, size_t q, size_t r)
{
  T B [r-p];                           // temp space for merged copy of A
  g_set_merge(A+p, A+q, A+q, A+r, B);  // merge the two parts of A to B
  g_copy(B, B+(r-p), A+p);             // copy B back to A[p,r)
}

By splitting the sort problem into halves [the "divide" step] we reduce the problem to two sub-problems to be solved recursively [the "conquer" step] plus a merge operation [the "combine" step]:

C = cost = (cost of division into sub-problems) + (cost of conquering sub-problems) + (cost of combining sub-solutions to solution)
C = cost = 0 + 2×C(n/2) + Θ(n)

Letting T(n) denote the runtime of the algorithm with input size n, we have just shown that T satisfies the recursion:

T(n) = 2×T(n/2) + Θ(n)

Let b(n) = B×n = Bn represent Θ(n) for some positive constant B. Note that k×b(n/k) = k×B×n/k = B×n = b(n). We can solve this recursion by substitution:

T(n) = 2×T(n/2) + b(n)
     = 2×(2×T(n/4) + b(n/2)) + b(n) = 4×T(n/4) + 2×b(n/2) + b(n) = 4×T(n/4) + b(n) + b(n) = 4×T(n/4) + 2b(n) 
     = ...
     = 2k×T(n/2k) + kb(n)

Assume for simplcity that n = 2^k and we have:

T(n) = 2k×T(n/2k) + b(n)×k 
     = n×T(1)  +  b(n)×log n
     = A×n + B×n×log n // where A = T(1)
     = An + B n log n
     = Θ(n log n)

We can also test a "guess" for compliance:

Lemma. The function f(n) = n log₂ n satisfies the recursion T(n) = 2×T(n/2) + n.

Proof. Calculate the right-hand side of the recursion:

2 f(n/2) + n = 2(n/2 log (n/2)) + n
             = n (log n/2 + 1)
             = n (log n - log 2 + 1)
             = n (log n - 1 + 1)
             = n log n
             = f(n)

which is the left-hand side. ∎

Conclusion. The asymptotic runtime of top-down merge sort is O(n log n).

2 Sidebar (bottom-up merge_sort)

void merge_sort_bu (A, n)
{
  if (n < 2) return;
  for (size_t i = 1; i < n; i = i+i)
  {
    for (size_t j = 0; j < n - i; j += i+i)
    {
      if (n < j+i+i) // possible at last step
        merge(A + j, A + j+i, A+n);
      else
        merge(A + j, A + j+i, A + j+i+i);
    }
  }
}

Proposition. The asymptotic runtime of bottom-up merge_sort is O(n log n).

Proof. The inner loop body is a call to merge on ranges of size i (at a cost of 2i). The inner loop runs n/2i times. Thus the cost of the inner loop is n, independent of i. (Another argument is that the inner loop touches each array element one time.)

The update rule for the outer loop doubles i and the stop rule terminates execution when i >= n. Thus the outer loop runs for each positive integer k until 2^k ≥ n, or equivalently k ≥ log n. Therefore the outer loop runs log n times. Therefore the runtime is n log n. ∎

Optimizations. Some very simple modifications to the straight algorithms can be made that reduce the runtime of the merge sorts on certain data sets. The simplest is the observation that if the last (largest) item in the first range is ≤ the first (smallest) item in the second range then the Θ(k) merge operation can be replaced with concatenation. (Here k is the length of the segments.) Depending on the data container, that operation is a straight data copy (vector - still Θ(k) but without any further comparisons) or a concatenation of link segments (list - just splice the entire pair of segments forward and skip the merge entirely - Θ(1)). This is why we stated the runtimes in terms of O rather than Θ.

3 FFT

The fast fourier transform (here in the context of multiplying two polynomials) is given by:

FFT (a, n)  // a is an array of size n (or smaller)
{
  if (n <= 1) return a;             // base case for recursion
  wn = exp(2*pi*i/n);               // wn = principal nth root of unity
  w  = 1;                           // w is maintained as wn^k, updated in post-processing loop
  b  = (a[0], a[2], ... , a[n-2]);  // even indexed values
  c  = (a[1], a[3], ... , a[n-1]);  // odd indexed values
  yb = FFT(b, n/2);                 // recursive call
  yc = FFT(c, n/2);                 // recursive call
  for (k = 0; k < n/2; ++k)
  {                               // loop invariant: w = wn^k at beginning of loop body
    ya[k]     = yb[k] + w*yc[k];  // see (1) below
    ya[k+n/2] = yb[k] - w*yc[k];  // see (2) below
    w = w*wn;                     // w = wn^(k+1) for next iteration
  }
  return ya;
}

This is clearly another example of a divide-and-conquer algorithm design. Looking at cost:

C = cost = (cost of division into sub-problems) + (cost of conquering sub-problems) + (cost of combining sub-solutions to solution)
C = cost = 0 + (cost of 2 recursive calls) + (cost of for loop)
C = cost = 2×C(n/2) + Θ(n)

We see the same recursion as for merge_sort. Therefore the same conclusion on runtime holds:

Proposition. The asymptotic runtime of FFT(n) is Θ(n log n).

(See [FFT Notes] and/or Chapter 30 of [Cormen] for more details.)

Exercise 1. Consider the two recursions:

R(n) = 2×R(n/2) + 1
S(n) = 2×S(n/2) + n²

Use substitution to show that R(n) = Θ(n) and S(n) = Θ(n²). (You may assume n = 2^k.)

4 Recurrences and the Master Theorem

Recurrences of the form

T(n) = aT(n/b) + f(n)

(where a ≥ 1 and b > 1 are real number constants) arise fairly often in the analysis of divide and conquer algorithms. Many of these are difficult to solve exactly in closed form. But there are results about the asymptotic behavior of solutions which are usually enough for the analysis. These are summarized by the:

Master Theorem. Assume the recurrence T as above and let d = log_b(a).

1. If f(n) ≤ O(n^d-ε) for some positive constant ε then T(n) = Θ(n^d).
2. If f(n) = Θ(n^d) then T(n) = Θ(n^d log n).
3. If f(n) ≥ Ω(n^d+ε) for some positive constant ε and if af(n/b) ≤ cf(n) almost everywhere for some constant c < 1 then T(n) = Θ(f(n)).

(See Section 4.5 of [Cormen] for a proof.)

Exercise 2. Use the Master Theorem and the results of Exercise 1 to show that the three recurrences:

R(n) = 2R(n/2) + 1
T(n) = 2T(n/2) + n
S(n) = 2S(n/2) + n²

illustrate the 3 cases of the theorem.

5^* Generating Functions

We will now illustrate the use of generating functions to solve some recurrences. This powerful technique is in general use in discrete mathematics as well as analysis of algorithms at the research level. The illustration here derives a closed form solution for the famous Fibonacci recursion (and is treated as a series of exercises in Chapter 4 of [Cormen]).

Given a sequence of numbers s₀, s₁, s₂, ... the (ordinary) generating functions for the sequence is the infinite sum

∑_ks_kz^k = s₀ + s₁z + s₂z² + ... + s_kz^k + ...

(Such sums are usually called formal power series.)^* Two such generating functions are equal iff their coefficients are equal. Let F denote the generating function for the Fibonacci sequence

F(z) = ∑_kf_kz^k = f₀ + f₁z + f₂z² + ... + f_kz^k + ...

where the coefficients f_k are given by the recursion:

f₀ = 0
f₁ = 1
f_k = f_k-1 + f_k-2, for k > 1

Straighforward calculation verifies that F satisfies the following:

F(z) = z + zF(z) + z²F(z)

To verify, look at the coefficients of z^k on the right-hand side.
   For k > 1: c_k = f_k-1 + f_k-2 is equal the coefficient f_k on the left, by the recursion.
   For k = 1: c₁ = 1 + f₀ = 1 = f₁
   For k = 0: c₀ = 0 = f₀ ∎

Now we can work some powerful mathematical magic, starting with algebra. Solving for F(z) we have:

z = F(z) - zF(z) - z²F(z)
z = (1 - z - z²)F(z)
F(z) = z / (1 - z - z²)

The quadratic denominator has real roots r₁, r₂ and hence can be factored as:

1 - z - z² = (-1)(z - r₁)(z - r₂) { r₁ and r₂ obtained with the "quadratic formula" }

Letting φ,ψ = (1 ± √5)/2 we can re-write as

1 - z - z² = (1 - φz)(1 - ψz) { verify by calculation }

Applying the technique of partial fractions we can obtain

F(z) = z / (1 - z - z²) = 1/√5 (1/(1 - φz) + 1/(1 - ψz)) { verify by calculation }

The final leap uses the identity

1/(1 - A) = ∑_kA^k

applied to each of the two fractions to obtain

F(z) = (1/√5)(∑_k φ^kz^k - ∑_k ψ^kz^k) = (1/√5)∑_k (φ^k - ψ^k)z^k

Equating coefficients we have proved:

Proposition. The kth Fibonacci number is given by the formula f_k = (φ^k - ψ^k)/√5.

The number φ = 1.61803... is called the golden ratio. Da Vinci used φ as the optimally esthetic length/width ratio for rectangles in architecture. Cormen observes that, because |ψ| < 1, f_k is equal to φ^k/√5 rounded to the nearest integer for k > 1.

^*Throughout this section ∑_k is taken to mean the sum over all non-negative integers k.