Midterm study guide

Everything from the eight slide decks, rewritten so it can be learned without the lecture videos. Each lecture page explains the ideas in plain language, works the examples on the slides step by step, answers every "Why?" the slides leave open, and lets you run the algorithms yourself.

What the course covers before the midterm

What an exam on these slides usually asks

Based on what the slides emphasize (and what CLRS-based midterms almost always include), expect questions of these kinds. Each one links to where it is taught.

A study plan if time is short

• Session 1Lectures 1–3. Learn the RAM model, how to count steps, and the three-part loop invariant proof. (about 1.5 h)
• Session 2Lecture 4a. This is the hardest and most tested material. Do every "Check yourself" and play with the constants widget until finding c and n₀ feels routine. (about 2 h)
• Session 3Lecture 4b. Memorize the log rules and summation formulas; rank functions by growth. (about 1 h)
• Session 4Lectures 5 and 6. Merge sort, recursion trees, master method. Solve at least ten recurrences with the solver, predicting the answer first. (about 2 h)
• Session 5Lectures 7 and 8. Trace selection sort, Shellsort, MaxHeapify, BuildMaxHeap and heapsort by hand, then check with the animations. (about 2 h)
• Session 6Practice exam, flashcards, cheat sheet. Re-read any page where you missed a question. (about 1.5 h)

The role of algorithms in computing

This first lecture sets up the vocabulary for the rest of the course: what a problem and an algorithm are, what it means to analyze one, and the big design strategies you will meet later. Expect definition-style questions from here.

Problems and algorithms

The slides separate two ideas that people often blur together.

Example: sorting is a problem (input: a sequence of numbers; output: the same numbers in non-decreasing order). Insertion sort, merge sort and heapsort are three different algorithms for that one problem.

Well-known computational problems on the slide

History, in one breath

Euclid (300 BC, the GCD algorithm below), Bhaskara (6th century), Al-Khwarizmi (9th century; the word "algorithm" comes from his name), Fibonacci (13th century), Babbage (19th century, mechanical computers), Turing (20th century, the formal model of computation), then von Neumann, Knuth, Karp, Tarjan and others.

Euclid's algorithm

The greatest common divisor GCD(a,b) is the largest integer dividing both. The slide builds the algorithm in two steps.

1. Observation. If a ≥ b, then GCD(a,b) = GCD(a−b, b). Any number dividing a and b also divides a−b, and the other way round, so the set of common divisors does not change.
2. Euclid's rule. Subtracting b over and over is the same as taking the remainder, so GCD(a,b) = GCD(a mod b, b). One mod replaces many subtractions.

What "analysis of algorithms" means

The slides list the basic issues around any algorithm: how to design it, how to express it (pseudocode), how to prove it correct, its efficiency (theoretical analysis on paper, or empirical analysis by running it) and its optimality (can anything do better?).

Correctness

Proving a big algorithm correct in one go is "dreadful", so the strategy is to break it into pieces, write each piece's precondition (what must be true before it runs) and postcondition (what it guarantees after), and prove each piece assuming the others are correct. Loop invariants (lecture 2 and 3) are exactly this idea applied to a loop.

Amount of work done

Instead of counting every instruction, pick the key instructions (for sorting: key comparisons) and count how often they run as a function of input size n. Then keep only the largest-growing term, written in asymptotic notation. Two kinds of analysis matter: worst case and average case.

Preview: the five growth-rate notations

Slide 16 defines them with limits. Lecture 4a covers them properly; for now just notice that each one says how f compares with g for huge n.

The other background the slide lists (floor and ceiling, exponentials and logarithms, Fibonacci numbers, summations and series) is all in Lecture 4b.

Algorithm design strategies

Slide 20 lists the classic strategies. You only need to recognize them now; several get whole lectures later.

Course roadmap

The whole semester, so you can see where the midterm sits. Everything up to heapsort is midterm material.

Check yourself

What is an algorithm, and how do we analyze one?

This lecture gives the machinery used for the rest of the course: a model of a computer (the RAM), a definition of running time (count the steps), the three kinds of case analysis, and the loop invariant technique for proving correctness. Linear search is the running example.

A sharper definition of "algorithm"

The slide lists five features every algorithm must have (Knuth's list). These are easy exam points if you can explain each in one line:

Formally, an algorithm can be defined as a Turing machine that always halts (lambda calculus is another equivalent definition). A mathematical definition is what lets us ask questions like "is this problem solvable at all?" and "which complexity class is it in?".

Running time and the RAM model

Algorithm analysis means predicting the resources an algorithm needs: time, memory, communication bandwidth. Time is the main concern. We analyze so we can pick the most efficient of several algorithms, know whether the best possible time is practical, and know whether something better could exist.

A running time has to be machine-independent, so we analyze on an idealized computer: the RAM (Random Access Machine) model.

Running time depends on the size of the input (number of items to sort; number of vertices and edges in a graph) and on other characteristics of the input (is it already sorted? does the graph have cycles?). That second dependency is why we need cases.

Worst, average and best case

Pseudocode conventions (CLRS p. 18–20)

• Indentation shows block structure (no braces).
• After a for loop ends, the loop counter keeps the value that made the test fail (after for j = 2 to n, j = n+1). Proofs rely on this.
• Parameters are passed by value; variables are local by default.
• Error handling and software-engineering concerns are left out.
• Arrays are indexed from 1: A[1..n].

Worked example: linear search

Input: a sequence of n numbers A[1..n] and a key. Output: true if the key occurs in the sequence, false otherwise.

Here x is how many times the loop test on line 2 runs. If the key is at position k, the test runs k times (it succeeds k−1 times, then stops on the match), so x = k. If the key is missing, the test runs for i = 1, …, n+1, so x = n+1. So x ranges from 1 to n+1, and the body (line 3) always runs one time fewer than the test.

Order of growth

What we really want is how running time grows with input size (order of growth), and the running time for large inputs (asymptotic complexity). For that, lower-order terms and the leading coefficient do not matter.

Notation preview (fully defined in lecture 4a): O(n), Θ(1), Ω(n²) and so on. Running time that does not depend on n at all is constant, written O(1) or Θ(1). For linear search: best case Θ(1), worst and average case Θ(n).

Comparing algorithms

Algorithm A is more efficient than B if A's complexity function has a lower order. The slide's pairs:

• Linear search Θ(n) versus binary search Θ(lg n).
• Insertion sort Θ(n²) versus quicksort Θ(n lg n) (on average) or merge sort Θ(nlg n).
• Multiplying two numbers: classical method O(nm); divide and conquer about O(n m^0.59). For 1000-digit operands: 40 s versus 15 s.
• Sorting 10⁶ numbers: insertion sort on a supercomputer in machine language took 5.56 hours; merge sort on a PC in C/C++ took 16.67 minutes. A better algorithm on a slow machine beats a worse algorithm on a fast machine, once n is large.

Why order of growth matters even as computers get faster

Hardware speed doubles every couple of years, so why care? Because the gain you get from faster hardware depends on the algorithm. Say the machine does S operations per second and you have one second. An n² algorithm handles the n with n² = S, so n = √S. Double S and n only grows by √2 ≈ 1.414. An n lg n algorithm almost doubles its n (gain 1.9). An exponential 2ⁿ algorithm gains only one extra item.

Correctness proofs

Proof techniques: construction, induction, contradiction. For loops, the standard tool is a loop invariant.

Correctness of linear search

Invariant: at the start of each iteration of the while loop, the key is not in the subarray A[1..i−1].

1. Initialization. Before the first iteration i = 1, so A[1..0] is empty. The key is trivially not in an empty subarray.
2. Maintenance. Suppose the key is not in A[1..i−1] and the loop body runs. The body only runs when the test passed, so A[i] ≠ key. Then the key is not in A[1..i]. After i is incremented, that is exactly "not in A[1..i−1]" for the new i.
3. Termination. The loop stops for one of two reasons. If i ≤ n, it stopped because A[i] = key, so returning true is correct. If i = n+1, the invariant says the key is not in A[1..n], the whole array, so returning false is correct.
4. It does terminate. i increases by 1 every iteration and the loop must stop once i > n, so there are at most n iterations.

The slide then says "go through the correctness proof of insertion sort in the text, p. 18–20". That is the whole of Lecture 3.

Check yourself

Insertion sort and loop invariants

A short deck with one job: show how a loop invariant proves an algorithm correct, using insertion sort. Being able to write this proof from memory is one of the most reliable exam points. The running-time analysis from CLRS 2.2 is included here too, since lecture 7 builds on it.

The algorithm

Problem: sort n numbers in A[1..n]. Output: A in sorted order, meaning A[i−1] ≤ A[i] for every i ∈ [2..n].

Idea: sort the way most people sort a hand of playing cards. The left part of the array is a sorted "hand". Take the next card (the key), slide every bigger card in the hand one place to the right, and drop the key into the gap.

Line 3 is a comment, as in CLRS, so the line numbers here match the cost table below.

Proof of correctness with a loop invariant

Both halves of that sentence matter. "Sorted" alone is not enough: you also need "the same elements that were there originally", otherwise an algorithm that overwrote everything with zeros would satisfy it.

1. Initialization. Before the first iteration j = 2, so the subarray is A[1..1]: the single original element A[1]. One element is always sorted. ✓
2. Maintenance. Assume A[1..j−1] holds the original first j−1 elements in sorted order. The while loop moves A[j−1], A[j−2], … one position to the right until it finds a position i with A[i] ≤ key (or reaches i = 0). Then the key, formerly A[j], goes into position i+1. Everything left of it is ≤ key and everything right of it (the shifted elements) is > key, so A[i] ≤ A[i+1] ≤ A[i+2]. Result: A[1..j] holds the original first j elements in sorted order. Incrementing j for the next iteration makes the invariant true again. ✓
   (Fully formal treatment would also state and prove an invariant for the inner while loop. The slides, like CLRS, accept this informal argument.)
3. Termination. The for loop ends when j exceeds length(A) = n, which means j = n+1. Substituting into the invariant: A[1..n] consists of the elements originally in A[1..n], in sorted order. That is the entire array, sorted. ✓

Running time analysis

CLRS gives each line a cost and counts how many times it runs. Let t_j be the number of times the while-loop test (line 5) runs for that value of j.

Line 1 runs n times, not n−1: the test runs once more than the body, when j = n+1 fails it.

Check yourself

Asymptotic notation: Θ, O, Ω, o, ω

The most important lecture before the midterm. These five symbols are the language for every running time in the course, and the exam will ask you both to use them and to prove statements with them by finding constants.

The idea in one paragraph

The asymptotic complexity of an algorithm is its running time as a function of input size n, for large n, written using only the highest-order term. Instead of "the running time is 3n² + 7n + 12 steps", we say "Θ(n²)". The notation describes how the function behaves in the limit, and it hides constant factors that depend on the machine anyway.

Each notation is defined as a set of functions. Θ(n²) is the set of all functions that grow like n². Writing f(n) = Θ(n²) really means f(n) ∈ Θ(n²). The slide says either form is accepted. All functions here are assumed non-negative for large n.

Θ-notation: tight bound

In words: from some point n₀ on, f is sandwiched between two constant multiples of g. g(n) is an asymptotically tight bound for f(n): same rate of growth.

Two things to notice. The constants only have to exist; you choose them. And nothing is promised for n < n₀: small inputs are ignored.

Worked example: prove 10n² − 3n = Θ(n²)

The slide's tip: make c₁ a little smaller than the leading coefficient, and c₂ a little bigger. To compare orders of growth, look at the leading term.

1. Upper bound. Pick c₂ = 10. Then 10n² − 3n ≤ 10n² for every n ≥ 1, because 3n ≥ 0.
2. Lower bound. Pick c₁ = 9. We need 9n² ≤ 10n² − 3n, i.e. 3n ≤ n², i.e. n ≥ 3.
3. Combine. With c₁ = 9, c₂ = 10, n₀ = 3: 0 ≤ 9n² ≤ 10n² − 3n ≤ 10n² for all n ≥ 3. ∎

Other choices work too (for example c₁ = 1, c₂ = 10, n₀ = 1). The exam only wants one valid set, with the algebra that justifies it.

O-notation: upper bound

Intuitively: the set of functions whose rate of growth is the same as or lower than g's. g(n) is an asymptotic upper bound for f(n). It need not be tight: n = O(n²) is true.

Since Θ demands both bounds, f(n) = Θ(g(n)) ⇒ f(n) = O(g(n)). As sets, Θ(g(n)) ⊆ O(g(n)).

Ω-notation: lower bound

Intuitively: functions whose rate of growth is the same as or higher than g's. g(n) is an asymptotic lower bound. Again f(n) = Θ(g(n)) ⇒ f(n) = Ω(g(n)), and Θ(g(n)) ⊆ Ω(g(n)).

How Θ, O and Ω relate

This is how tight bounds are usually proved in practice: prove an upper bound and a lower bound separately, then combine them.

What O, Ω and Θ say about running times

This slide (asymp-15) is subtle and a favourite source of true/false questions.

Asymptotic notation inside equations

A Θ inside a formula stands for some anonymous function in that set. The slide's chain:

How to interpret it (the slide asks): in the second expression, Θ(n) stands for the specific function 2n + 1, which is in Θ(n). In the third, Θ(n²) stands for 3n² + 2n + 1. Each "=" is read left to right as "no matter how the anonymous functions on the left are chosen, there is a way to choose the ones on the right so the equation holds". The chain shows we can throw away lower-order terms one at a time.

The same convention lets us write recurrences like T(n) = 2T(n/2) + Θ(n).

o-notation and ω-notation: strict bounds

More examples: n²2 = ω(n), but n²2 ≠ ω(n²). And n lg n = o(n²), n² = ω(nlg n).

The real-number analogy

Comparing functions behaves like comparing two numbers a and b:

Using limits: the fastest way to compare

When the limit of f(n)/g(n) exists, it decides the relationship immediately. This is usually quicker than hunting for constants.

L'Hôpital's rule is the main tool: if both go to ∞, lim fg = lim f′g′. Example, lg n versus √n: (lg n)′(√n)′ = 1/(n ln 2)1/(2√n) = 2ln 2 · √n → 0, so lg n = o(√n).

Properties

Check yourself

Common functions and summations

The algebra toolkit. Log and exponent rules let you compare growth rates; summation formulas turn loops into closed-form running times. Most of this is memorization, and the cheat sheet repeats all of it.

Monotonicity, floors and ceilings

Floors appear all over the course: merge sort splits at q = ⌊ (p+r)/2 ⌋, a heap node's parent is ⌊ i/2 ⌋, a heap has height ⌊ lg n ⌋. A polynomial of degree d with positive leading coefficient is Θ(n^d).

Exponentials

Logarithms

x = log_b a is the exponent with a = b^x. Notation: ln a = log_e a (natural), lg a = log₂ a (binary, the course default), lg^k a = (lg a)^k (a power, not repeated log), lglg a = lg(lg a) (composition).

Polylogarithms and factorials

The growth-rate ladder

From slowest to fastest. Each function is o of the next one. Memorize this order; "rank these functions" is a classic exam question.

The slide's exercise, fully worked

"Express functions in A in asymptotic notation using functions in B."

Summations: why we need them

A loop's running time is a sum: add up the cost of each iteration. Nested loops give nested sums. The slides' example is the maximum subvector problem.

Count how many times line 6 runs:

The slide says "this is not a simplified solution. What is the final answer?" Simplify from the inside out:

1. Innermost: j∑k=i 1 = j − i + 1 (constant series: count of terms).
2. Middle: n∑j=i (j − i + 1) = 1 + 2 + ⋯ + (n−i+1) = (n−i+1)(n−i+2)2 (arithmetic series).
3. Outer: substitute p = n − i + 1, which runs from n down to 1: n∑p=1p(p+1)2 = 12(∑ p² + ∑ p) = 12(n(n+1)(2n+1)6 + n(n+1)2).
4. Simplify: = n(n+1)12(2n + 1 + 3) = n(n+1)(n+2)6.

The slide then shows two faster solutions from Wikipedia's "Maximum subarray problem" page:

Summation formulas to memorize

Approximation by integrals

When there is no closed form, bound the sum by integrals. For monotonically increasing f:

For monotonically decreasing f the inequalities flip:

How? (the slide asks) Draw each term f(k) as a rectangle of width 1. For a decreasing function, the rectangle on [k, k+1] with height f(k) sits above the curve, so the rectangles' total area (the sum) is at least the area under the curve. Shift the rectangles one unit left, to [k−1, k], and they sit below the curve, giving the upper bound.

From loops to sums: a quick reference

Check yourself

Divide and conquer: merge sort and recurrences

Merge sort is the model divide-and-conquer algorithm. Analyzing it produces a recurrence, and the second half of the lecture is about solving recurrences three ways: substitution, recursion trees, and (introduced here, covered in lecture 6) the master method.

The paradigm

Divide-and-conquer algorithms are recursive in structure. Each level of recursion does three things:

For merge sort on n elements: divide the sequence into two halves of n/2 elements; conquer by sorting each half recursively with merge sort; combine by merging the two sorted halves. The base case is a sequence of length 1, which is already sorted. All the real work happens in the combine step.

Merge sort

The slide's example sorts ⟨ 18, 26, 32, 6, 43, 15, 9, 1⟩. The left half of the picture is the divide phase splitting down to single elements; the right half is the merges climbing back up. Step through it:

The Merge procedure

Input: an array where A[p..q] and A[q+1..r] are each already sorted. Output: A[p..r] sorted.

Lines 1–6 copy the two halves into new arrays L and R. Lines 11–16 repeatedly compare the smallest remaining element of each (L[i] and R[j]), copy the smaller one back into A, and advance that side's pointer. Think of two sorted piles of cards face up: always take the smaller top card.

Correctness of Merge

1. Initialization. Before the first iteration k = p, so A[p..k−1] is empty: it trivially holds the 0 smallest elements. i = j = 1, and L[1], R[1] are the smallest elements of their (sorted) arrays, none copied yet.
2. Maintenance. Case L[i] ≤ R[j]: by the invariant, L[i] is the smallest element not yet copied. Line 13 puts it in A[k], so A[p..k] now holds the k − p + 1 smallest, sorted. Incrementing i (line 14) and k (the for loop) re-establishes the invariant. The case L[i] > R[j] is symmetric (lines 15–16).
3. Termination. The loop ends with k = r + 1. By the invariant, A[p..r] holds the r − p + 1 smallest elements of L and R, sorted. L and R together hold n₁ + n₂ + 2 = r − p + 3 elements, so everything except the two sentinels has been copied back.

Analysis of merge sort

Let T(n) be the worst-case running time on n elements (assume n is a power of 2, so halves are exact).

• Divide: computing the middle q takes Θ(1).
• Conquer: two subproblems of size n/2 take 2T(n/2).
• Combine: merging n elements takes Θ(n).

The rest of this lecture shows that T(n) = Θ(n lg n).

Recurrences

A general divide-and-conquer algorithm that makes a subproblems, each 1/b the size, with divide cost D(n) and combine cost C(n):

Three solution methods (CLRS Chapter 4): substitution, recursion tree, master method.

Method 1: substitution

Guess the form of the solution, then prove it by mathematical induction. The name comes from substituting the guessed answer for the function when applying the inductive hypothesis to smaller values. It works well when the solution is easy to guess, and there is no general way to guess.

Method 2: recursion trees

A good guess can be hard to come by. A recursion tree draws out the recurrence: each node is the cost of one subproblem (not counting its recursive calls), children are its subproblems. Sum each level, then sum the levels. It organizes the algebra and produces a guess you can verify by substitution.

For merge sort, rewrite the recurrence with an explicit constant c > 0 (the base-case time, and the per-element time for divide plus combine): T(1) = c, T(n) = 2T(n/2) + cn.

• Each level costs cn: going down one level doubles the number of subproblems but halves the cost of each.
• Level i has 2ⁱ nodes of size n/2ⁱ; the leaves are at size 1, so the bottom level is i = lg n. That gives lg n + 1 levels (height lg n, assuming n is a power of 2; provable by induction).
• Total = (number of levels) × (cost per level) = (lg n + 1) cn = cnlg n + cn = Θ(nlg n).

The slide's two "other examples", solved

Method 3: the master method (preview)

A "cookbook" for recurrences of the form T(n) = aT(n/b) + f(n) with constants a ≥ 1, b > 1 and f asymptotically positive. It compares f(n) with n^{log_b a} and has three cases to memorize. Lecture 6 covers it in full, with a solver you can practise on.

Check yourself

Recurrences II: the master method

A cookbook that solves most divide-and-conquer recurrences in three lines. Memorize the three cases, understand what each one means in the recursion tree, and know the situations where the theorem does not apply. Almost every algorithms midterm has at least one of these.

The master theorem

It applies to recurrences of the form

where a ≥ 1 and b > 1 are constants and f(n) is asymptotically positive. n/b may stand for ⌊ n/b ⌋ or ⌈ n/b ⌉.

Everything hinges on one comparison: f(n) versus n^{log_b a}. Call n^{log_b a} the watershed function. The bigger of the two wins; if they tie, you pick up an extra lg n.

Why it works: the recursion tree

Draw the tree for T(n) = aT(n/b) + f(n). The root costs f(n) and has a children; each child costs f(n/b) and has a children, and so on.

• Level j has a^j nodes, each of size n/b^j, costing a^j f(n/b^j) in total.
• Sizes hit 1 when n/b^j = 1, i.e. at depth j = log_b n.
• Number of leaves = a^{log_b n} = n^{log_b a} (by the log swap rule). Each costs Θ(1), so the leaves cost Θ(n^{log_b a}).

That is why n^{log_b a} is the thing to compare with: it is the total cost of the leaves. The three cases are the three ways that sum can behave.

You can see this in the recursion tree explorer on lecture 5: the level-cost bars grow, stay flat, or shrink.

Applying it: a recipe

1. Check the form: constant a ≥ 1, constant b > 1, all subproblems the same size n/b.
2. Compute n^{log_b a}. Simplify the exponent when you can (log₂ 4 = 2, log₄ 16 = 2, log₃ 9 = 2); otherwise leave it, e.g. n^{log₂ 3} ≈ n^1.585.
3. Compare f(n) with it.
   • Smaller by a factor of n^ε? Case 1.
   • Same up to a constant? Case 2.
   • Larger by a factor of n^ε? Check regularity (a f(n/b) ≤ c f(n) with c < 1), then case 3.
   • Larger or smaller but only by a log factor? The theorem does not apply.
4. Write the answer.

The slide's four examples

More examples (the kind you will see on the exam)

When the master theorem does not apply

• Subproblems of different sizes, like T(n/3) + T(2n/3) + n. Use a recursion tree.
• Not a division, like T(n−1) + n (that is insertion sort's recursive form). Unroll it: n + (n−1) + ⋯ + 1 = Θ(n²).
• a not a constant ≥ 1 or b not a constant > 1, like 2ⁿ T(n/2) + n or 0.5 T(n/2) + n.
• The gap: f(n) larger or smaller than n^{log_b a} but not polynomially, like nlg n or n/lg n against n.
• Case 3 without regularity. Rare for the polynomial-type f you will see, because for f(n) = n^k with k > log_b a regularity is automatic: a f(n/b) = ab^kn^k and ab^k < 1 exactly when k > log_b a.

Check yourself

Elementary sorting algorithms

The vocabulary of sorting (stable, in place, comparison-based), two simple Θ(n²) algorithms analyzed carefully, a theorem explaining why they are stuck at n², and Shellsort, which escapes by fixing many inversions per comparison.

Definitions

Input: n records R₁, …, R_n. Each record has a key K_i and possibly other satellite information that travels with it (think: sorting student records by ID).

The keys need an ordering relation ≤ with two properties:

Such a relation is a total ordering (linear ordering) on the keys.

Terminology (likely exam material)

Categories of sorting

Why study elementary methods? They show the basic mechanisms clearly, they are good for small files and for well-structured files (almost sorted), and they are used inside more powerful methods (e.g. finishing off small subarrays in quicksort or merge sort).

Selection sort

The slide's version builds the sorted part at the right end: for i = n down to 2, find the largest element in A[1..i] and swap it into position i.

Insertion sort, analyzed more carefully

Same algorithm as lecture 3 (the slide writes the test as key < A[i], which is the same thing). It is in place and stable.

Worst case

Inserting A[i] takes at most i − 1 comparisons, so

For which input does it perform exactly n(n−1)/2 comparisons? A strictly decreasing (reverse-sorted) array: every new key is smaller than everything before it and travels all the way to position 1.

Average case

Assume all orderings are equally likely. Then the key A[j] is equally likely (probability 1/j) to belong in any of the positions 1..j of the sorted prefix.

• If it belongs at position k with 2 ≤ k ≤ j, the while loop compares it with A[j−1], …, A[k−1]: that is j − k + 1 comparisons (the last one fails and stops the loop).
• If it belongs at position 1, it is compared with all j − 1 elements and the loop stops because i reaches 0, not because of a failed comparison: j − 1 comparisons.

Summing over all keys:

So on average insertion sort does about half the worst-case work, n²/4, which is still quadratic.

Inversions: why simple sorts are stuck at n²

A sorted array has 0 inversions. Sorting means removing all of them.

• Worst case: n(n−1)2 inversions, for the reverse-sorted permutation ⟨ n, n−1, …, 1⟩ (every pair is out of order).
• Average case: n(n−1)4. The argument: let Π^T be Π reversed, ⟨Π_n, …, Π₁⟩. For each of the n(n−1)2 pairs, the pair is an inversion in exactly one of Π and Π^T (reversing flips the relative order). So inv(Π) + inv(Π^T) = n(n−1)2. Permutations come in such pairs, so the average is half: n(n−1)4.

So if we want to beat Θ(n²), each comparison has to remove more than a constant number of inversions. That means moving elements over long distances, which is exactly what Shellsort, merge sort, heapsort and quicksort do.

Shellsort

A simple extension of insertion sort that gains speed by letting elements that are far apart be exchanged, fixing many inversions with one move.

• To h-sort the file: split it into h interleaved subsequences, each made of keys h positions apart, and insertion-sort each one. The result is h-sorted: taking every h-th key starting anywhere gives a sorted sequence.
• h-sort for a decreasing sequence of increments, ending with h = 1.
• Large-h passes are cheap (short subsequences) and move elements long distances, so later small-h passes see a file that is nearly sorted, where insertion sort is fast.
• Correctness is easy: the last pass (h = 1) is plain insertion sort, which sorts anything.

Shellsort is really a family of algorithms, one for each increment sequence {h_k}. The slide's code uses h ← 3h + 1, which gives 1, 4, 13, 40, 121, … (Knuth's sequence).

The slide writes for i = h to n and if j < h then break, which is correct with 0-based array indices. With 1-based indices the loop starts at h + 1 and the guard is j ≤ h, as shown. When h = 1 this is exactly insertion sort; otherwise it is insertion sort on keys h apart.

Summary of the sorts so far

Check yourself