Algorithm

• Body
• Recipe for Computation
• Input
• Output
• Book and common practice: equate "algorithm" with "algorithm.body"
• Assumptions
• Conclusions
• Proof(s)
• Process halts
• If (assumptions) and (algorithm.body is executed) then (conclusions)
• Analysis (optional)
• Time complexity
• Space complexity
• Other properties (such as "in place" or "stable" for a sorting algorithm)
• Proofs of analytic conclusions

Algorithm Concepts

• Computational Problem
• Input
• Instance of Input
• Example:
  • Sorting Problem: Given a sequence of n numbers, find a permutation such that the order is non-decreasing.
  • Input: A = (a₁, a₂, ... , a_n)
  • Output: A' = (a'₁, a'₂, ... , a'_n) such that a₁ <= a₂ <= ... <= a_n
  • Input Instance: A = (10, 5, 15, 10, 25, 20)
  • Output: A' = (5, 10, 10, 15, 20, 25)
• Data Structures - often part of assumptions
• NP-Complete Problems
• Approximation Algorithms

Insert Sort

Analysis of Insertion Sort

                                     incr cost   times
Insert-Sort ( array of numbers A )   ---------   -----
{
  for (j = 2; j <= length(A); ++j)       c1        n
  {
    key = A[j];                          c2        n-1
    i = j - 1;                           c3        n-1
    while (i > 0 and A[i] > key)         c4        SUM[2,n] tj
    {
      A[i+1] = A[i];                     c5        SUM[2,n] (tj -1)
      i = i-1;                           c6        SUM[2,n] (tj -1)
    }
    A[i+1] = key;                        c7        n-1
  }
  return;
}

where t_j is the number of times the while loop header is executed for that particular j (dependent on input data instance).

Total Cost = c₁n + c₂(n-1) + c₃(n-1) + c₄Σ₂ⁿ t_j + c₅Σ₂ⁿ (t_j-1) + c₆Σ₂ⁿ (t_j-1) + c₇ (n-1)

Best Case Cost = (c₁ + c₂ + c₃ + c₄ + c₇)n - (c₂ + c₃ + c₄ + c₇) = An + B
for some constants = A and B.

Worst Case Cost = an² + bn + c for some constants a, b, and c

Asymptotic Notation

• Big O:
  g(n) <= O(f(n)) iff there exist positive constants c and n₀ such that 0 <= g(n) <= cf(n) for all n >= n₀
  "g is asymptotically bounded above by f "
• Big Omega:
  g(n) >= Ω(f(n)) iff there exist positive constants c and n₀ such that 0 <= cf(n) <= g(n) for all n >= n₀
  "g is asymptotically bounded below by f "
• Big Theta:
  g(n) = Θ(f(n)) iff there exist positive constants c₁, c₂, and n₀ such that 0 <= c₁f(n) <= g(n) <= c₂f(n) for all n >= n₀
  "g is asymptotically bounded above and below by f "

Theorem 1 (transitivity).
(1) If f(n) is in O(g(n)) and g(n) is in O(h(n)) then f(n) is in O(h(n))
(2) If f(n) is in Ω(g(n)) and g(n) is in Ω(h(n)) then f(n) is in Ω(h(n))
(3) If f(n) is in Θ(g(n)) and g(n) is in Θ(h(n)) then f(n) is in Θ(h(n))

Theorem 2 (anti-symmetry). f(n) is in O(g(n)) iff g(n) is in Ω(f(n)).

Theorem 3 (symmetry). f(n) is in Θ(g(n)) iff g(n) is in Θ(f(n)).

Theorem 4 (reflexivity). A function is asymptotically related to itself: f(n) is in O(f(n)), f(n) is in Ω(f(n)), and f(n) is in Θ(f(n)).

In particular, of the three, Θ defines an equivalence relation on functions while O and Ω define an anti-symmetric pair of relations on functions that is analogous to the pair of order relations (<=, >=) on numbers. Thus, Θ is analogous to '=' (equality), while O is analogous to '<=' (less than or equal to) and Ω is analogous to '>=' (greater than or equal to). There is even a form of the dichotomy property of order relations:

Theorem 5 (dichotomy). If f(n) <= O(g(n)) and g(n) <= O(f(n)) then f(n) = Θ(g(n)).

All of these theorems can be restated in an equivalent form using set notation. For example, Theorem 5 restates as follows:

Theorem 5a (dichotomy). If O(f(n)) subset_of O(g(n)) and O(g(n)) subset_of O(f(n)) then Θ(f(n)) = Θ(g(n))).

We will use the notation of equality and inequality, as shown in the slide, as a notational device that helps reinforce our perception of the nature of these three relations. (We recognize that this is an abuse of notation. See the discussion of notation "abuse" and notation "misuse" in [Cormen].)

Insertion Sort Asymptotics

• Best Case Runtime = Θ(n), n = size of input
• Worst Case Runtime = Θ(n²), n = size of input
• Average Case Runtime = Θ(n²), n = size of input