AlgNotes 1

FSU Seal - 1851

COT 5405
Advanced Algorithms
Chris Lacher
Notes 1: Intro

Algorithm

Body

Recipe for Computation
Input
Output
Book and common practice: equate "algorithm" with "algorithm.body"

Assumptions
Conclusions
Proof(s)

Process halts
If (assumptions) and (algorithm.body is executed) then (conclusions)

Analysis (optional)

Time complexity
Space complexity
Other properties (such as "in place" or "stable" for a sorting algorithm)
Proofs of analytic conclusions

Algorithm Concepts

Computational Problem
Input
Instance of Input
Example:
- Sorting Problem: Given a sequence of n numbers, find a permutation such that the order is non-decreasing.
- Input: A = (a₁, a₂, ... , a_n)
- Output: A' = (a'₁, a'₂, ... , a'_n) such that a₁ <= a₂ <= ... <= a_n
- Input Instance: A = (10, 5, 15, 10, 25, 20)
- Output: A' = (5, 10, 10, 15, 20, 25)
Data Structures - often part of assumptions
NP-Complete Problems
Approximation Algorithms

Insert Sort

Insert-Sort ( array of numbers A )
{
  for (j = 2; j <= length(A); ++j)
  {
    // Loop Invariant: A[1..j-1] is sorted
    key = A[j];
    // insert A[j] into A[1..j-1]
    i = j - 1;
    while (i > 0 and A[i] > key)
    {
      A[i+1] = A[i];
      i = i-1;
    }
    A[i+1] = key;
  }
  return;
}

Initialization
LI[2] true
Maintenance
LI[j-1] implies LI[j]
Conclusion
LI[n] proves correctness

Analysis of Insertion Sort

                                     incr cost   times
Insert-Sort ( array of numbers A )   ---------   -----
{
  for (j = 2; j <= length(A); ++j)       c1        n
  {
    key = A[j];                          c2        n-1
    i = j - 1;                           c3        n-1
    while (i > 0 and A[i] > key)         c4        SUM[2,n] tj
    {
      A[i+1] = A[i];                     c5        SUM[2,n] (tj -1)
      i = i-1;                           c6        SUM[2,n] (tj -1)
    }
    A[i+1] = key;                        c7        n-1
  }
  return;
}
where t_j is the number of times the while loop header is executed for that particular j (dependent on input data instance).

Observations: The value of t_j-1 is the number of elements of A[1..j-1] that are greater than key, obviously a data-dependent value.

In the best case, the inner loop body is never executed. Therefore t_j = 1.

In the worst case, the inner loop body is executed j-1 times. Therefore t_j = j.

For arbitrary data, there is equal liklihood that key falls in a particular place in A[1..j-1]. Therefore the inner loop body is executed an average of (j -1)/2 times, so the average value of t_j is j /2.

Total Cost = c₁n + c₂(n -1) + c₃(n -1) + c₄Σ₂ⁿ t_j + c₅Σ₂ⁿ (t_j -1) + c₆Σ₂ⁿ (t_j -1) + c₇ (n -1)

Best Case Cost
= (c₁ + c₂ + c₃ + c₄ + c₇)n - (c₂ + c₃ + c₄ + c₇)
= An + B, for some constants A and B.

Worst Case Cost
= c₁n + c₂(n -1) + c₃(n -1) + c₄Σ₂ⁿ j + c₅Σ₂ⁿ (j -1) + c₆Σ₂ⁿ (j -1) + c₇ (n -1)
= An² + Bn + C for some constants A, B, and C (using the fact that Σ₁ⁿ j = n(n -1)/2)

Average Case Cost
= c₁n + c₂(n -1) + c₃(n -1) + c₄Σ₂ⁿ (j /2) + c₅Σ₂ⁿ (j /2 -1) + c₆Σ₂ⁿ (j /2 -1) + c₇ (n -1)
= An² + Bn + C for some constants A, B, and C (using the fact that Σ₁ⁿ (j /2) = n(n -1)/4)

Asymptotic Notation

Big O:
g(n) <= O(f(n)) iff there exist positive constants c and n₀ such that 0 <= g(n) <= cf(n) for all n >= n₀
"g is asymptotically bounded above by f "

Big Omega:
g(n) >= Ω(f(n)) iff there exist positive constants c and n₀ such that 0 <= cf(n) <= g(n) for all n >= n₀
"g is asymptotically bounded below by f "

Big Theta:
g(n) = Θ(f(n)) iff there exist positive constants c₁, c₂, and n₀ such that 0 <= c₁f(n) <= g(n) <= c₂f(n) for all n >= n₀
"g is asymptotically bounded above and below by f "

Theorem 1 (transitivity).
(1) If f(n) is in O(g(n)) and g(n) is in O(h(n)) then f(n) is in O(h(n))
(2) If f(n) is in Ω(g(n)) and g(n) is in Ω(h(n)) then f(n) is in Ω(h(n))
(3) If f(n) is in Θ(g(n)) and g(n) is in Θ(h(n)) then f(n) is in Θ(h(n))

Theorem 2 (anti-symmetry). f(n) is in O(g(n)) iff g(n) is in Ω(f(n)).

Theorem 3 (symmetry). f(n) is in Θ(g(n)) iff g(n) is in Θ(f(n)).

Theorem 4 (reflexivity). A function is asymptotically related to itself: f(n) is in O(f(n)), f(n) is in Ω(f(n)), and f(n) is in Θ(f(n)).

In particular, of the three, Θ defines an equivalence relation on functions while O and Ω define an anti-symmetric pair of relations on functions that is analogous to the pair of order relations (<=, >=) on numbers. Thus, Θ is analogous to '=' (equality), while O is analogous to '<=' (less than or equal to) and Ω is analogous to '>=' (greater than or equal to). There is even a form of the dichotomy property of order relations:

Theorem 5 (dichotomy). If f(n) <= O(g(n)) and g(n) <= O(f(n)) then f(n) = Θ(g(n)).

All of these theorems can be restated in an equivalent form using set notation. For example, Theorem 5 restates as follows:

Theorem 5a (dichotomy). If O(f(n)) subset_of O(g(n)) and O(g(n)) subset_of O(f(n)) then Θ(f(n)) = Θ(g(n))).

We will use the notation of equality and inequality, as shown in the slide, as a notational device that helps reinforce our perception of the nature of these three relations. (We recognize that this is an abuse of notation. See the discussion of notation "abuse" and notation "misuse" in [Cormen].)

Insertion Sort Asymptotics

Best Case Runtime = Θ(n), n = size of input

Worst Case Runtime = Θ(n²), n = size of input

Average Case Runtime = Θ(n²), n = size of input