Sorting Problem

• Input: Sequence of n numbers or keys (a₀, a₁, ... , a_n-1)
• Output: Permutation (a'₀, a'₁, ... , a'_n-1) such that a'₁ <= a'₂ <= ... <= a'_n
• Satellite data associated with keys may be considerable
• Implementation "details":
• Assume that (!(key1 < key2) and !(key2 < key1)) implies (key1 == key2), but satellite data may not be the same
• May use indirection to avoid many re-assignments of satellite data
• Data assumed in random access storage (e.g., array); sorting algorithm may require copy into such a structure

Insertion Sort

InsertionSort ( array of numbers A , length n )
{
  for (i = 1; i < n; ++i)
  {
    // Loop Invariant: A[0..i) is sorted
    t = A[i];
    for (j = i; j > 0 && t < A[j - 1]; --j)
      A[j] = A[j - 1];
    A[j] = t;
  }
  return;
}

• Proof of Halting (done)
• Proof of Correctness (done)
• Runtime Analysis (done), Result: Worst case = Θ(n²), Average Case = Θ(n²), Best Case = Θ(n)
• Runspace Analysis (done), Result: in-place

Selection Sort

inline Swap (key& x, key& y)
{
  key z = x;
  x = y;
  y = z;
}

SelectionSort ( array A[0..n) )
// pre: 
// post: A[0..n) is sorted
{
  for (i = 0; i != n; ++i)
  {
    // Loop Invariant: A[0..i) is sorted
    k = i;
    for (j = i; j != n; ++j)
      if (A[j] < A[k])
        k = j;
    // Loop Invariant: A[k] is the first smallest element of A[i..n)
    Swap (A[i], A[k]);
  }
  return;
}

• Proof of Halting
• Proof of Correctness
• Runtime Analysis
• Runspace Analysis
• Is selection sort stable?
• Is insertion sort stable?

Merge Sort

Merge(array A, index p, index q, index r)
// Pre:  A[p..q) and A[q..r) are sorted ranges
// Post: A[p..r) equals merge of A[p..q) and A[q..r)
{
  T B [r-p];                                // temp space for merged copy of A
  fsu::g_set_merge(A+p, A+q, A+q, A+r, B);  // merge the two parts of A to B
  fsu::g_copy(B, B+(r-p), A+p);             // copy B back to A[p,r)
  return;
}

MergeSort (array A, index p, index r)
// Pre:  [p..r) is a subrange of A
// Post: A[p..r) is sorted
{
  if (r - p > 1)
  {
    q = (p+r)/2;         // integer arithmetic
    MergeSort(A,p,q);    // recursive call
    MergeSort(A,q,r);    // recursive call
    Merge(A,p,q,r);      // defined above
  }
  return;
}

• Proof of Halting
• Proof of Correctness
• Runtime Analysis
• Runspace Analysis

HeapSort

• Binary tree model
from array index structure:
• A max heap is an array in which every parent is greater than or equal to its children, using the tree model described above. (This is also called the partially ordered tree (POT) property.)
• Review Heaps and Heap Algorithms

HeapSort (array A, index p index r)
// pre:  p and r are valid index values for A
// post: A[p .. r] is sorted
{
  for (i = p+1; i <= r; ++i)
    push_heap(A,p,i)
  for (i = r; i > p; --i)
    pop_heap(A,p,i)
}

• Proof of Halting
• Proof of Correctness
• Runtime Analysis
• Runspace Analysis
• Recursive version?

QuickSort

QuickSort(array A, index p, index r)
// Pre:  A is an array of type T 
//       A is defined for the range [p,r)
// Post: A[p,r) is sorted
{
  if (r - p > 1)
  {
    q = Partition(A,p,r);
    QuickSort(A,p,q);
    QuickSort(A,q+1,r);
  }
  return;
}

index Partition(array A, index p, index r)
{
  i = p;
  for (j = p; j < r-1; ++j)
  // loop invariants:
  //  1. if k is in [p..i) then A[k] <= A[r-1]
  //  2. if k is in [i..j) then A[k] > A[r-1]
  {
    if (A[j] <= A[r-1]) // if A[j] <= last
    {
      Swap(A[i],A[j]);    // swap to low range
      ++i;                // expand the low range
    }
  }
  Swap(A[i],A[r-1]);    // swap last into pivot location
  return i;
}

• Proof of Halting
• Proof of Correctness
• Worst Case Runtime O(n²)
• Average Case Runtime Θ(n log n)

Proof:

1. An element is used as a pivot at most one time, so there are at most n calls made to the partition routine.

2. All comparisons are made inside the partition. Therefore

   Quicksort runtime <= O(n + x)

   where x is the total number of comparisons made by the partition routine.

3. Consider the case of sorted input. Then every element will serve as a pivot and every other element will be compared to it. Thus the partition routine will be called at least n times and the k-th call will perform k-1 comparisons. Therefore

   Worst case runtime >= Ω(n²)

   Definitions. In order to estimate the average case runtime, define the following entities:

   E[x] = expected value of x
   z₀, z₁, ..., z_n-1 = the elements in sorted order
   [z_i,z_j] = {z_i,z_i+1 ... z_j}
   x_i,j = bool{z_i is compared to z_j}
   e_i,j = Probability{z_i is compared to z_j}
   

4. No pair is compared more than one time. Therefore

   x = Σ_0..n-2 Σ_{j = i+1..n-1} x_i,j = ΣΣ_i,j x_i,j
   

   where the double sum ranges over the upper triangle of indices defined by 0 <= i < j <= n-1.

5. Compute the expected value:

   E[x]	= E[ΣΣi,j xi,j]
   	= ΣΣi,j E[xi,j]
   	= ΣΣi,j ei,j

6. Because of the way partition divides the data using the pivot:

   1. If a pivot p is selected with z_i < p < z_j then z_i and z_j will never be compared at a subsequent stage of the process.
   2. If z_i is selected as a pivot before any other item in [z_i,z_j] then every element in (z_i,z_j] will be compared to z_i
   3. If z_j is selected as a pivot before any other item in [z_i,z_j] then every element in [z_i,z_j) will be compared to z_j

   It follows that:

   z_i is compared to z_j iff the first element in [z_i,z_j] to be chosen as a pivot is one of the two ends of the interval.

7. Assuming pivot values are chosen at random, we have:

   ei,j	= P{zi or zj is first chosen from [zi,zj]}
   	= P{zi is first chosen from [zi,zj]} + P{zj is first chosen from [zi,zj]}
   	= 1/(j - i + 1) + 1/(j - i + 1)
   	= 2/(j - i + 1)

8. Estimate the expected value as follows:

   E[x]	= Σ0..n-2 Σj = i+1..n-1 ei,j
   	= Σ0..n-2 Σj = i+1..n-1 2/(j - i + 1) [substitute k = j - i]
   	= Σ0..n-2 Σk = 1..n-i 2/(k + 1)
   	< Σ0..n-1 Σk = 0..n-1 2/(k + 1)
   	= Σ0..n-1 O(log n)
   	= O(n log n)

   Putting all these observations together completes the argument. Note that the assumption that any element in the interval [z₀,z_n) is equally likely to be chosen as a pivot is used in observation 7.

• Runspace Analysis

Theoretical Limits

• Theorem: Any comparison sort algorithm has worst case runtime Ω(n log n).
1. Path in decision tree from root to leaf represents one particular sort instance
2. Therefore: Worst case runtime = Ω(H)
3. Decision tree must have each possible permutation of input as a reachable leaf
4. Therefore: Decision tree has L >= n! leaves
5. n! <= L <= 2^H
6. log n! <= H
7. H >= Ω (log n!) >= Ω (n log n) (by "formula")