Probability and Statistics for Engineering Lecture 1-2 Notes

Basic Ideas in Probability

Basic Concepts

Definition: Random Experiment

A random experiment, often simply called an experiment, represents the realization or observation of a random phenomenon and has the following characteristics:

• it can be repeated under the same conditions;
• all possible outcomes are clearly known;
• exactly one of these possible outcomes occurs each time, but it cannot be determined in advance which outcome will occur.

Definition: Sample Point and Sample Space

Each possible fundamental outcome of a random experiment is called a sample point, usually denoted as . A set that includes all sample points of the experiment is called the sample space, usually denoted as .

Definition: Event

From a set theory perspective, a random event, or simply event, is a subset of the sample space , typically denoted by uppercase letters. If the outcome is a sample point in event , then we say that event happens.

• Inclusion : happens when happens.
• Sum/union /: at least one of and happens.
• Product/intersection /: and both happen.
• Difference /: happens and does not happen.
• Mutually exclusive/disjoint : and cannot happen at the same time.
• Complement /: either or happens, denoted as or .

The operations of events obey certain rules similar to the rules of set:

• Communicative laws;
• Associative laws;
• Distributive laws;
• De Morgan’s laws:

Having defined the sample space and random event, we can now discuss the probability of events:

Definition: Probability

Probability measure, or simply probability, is a real-valued function defined on subsets of the sample space , satisfying the following three axioms: - Non-negativity: . - Normalization: . - Additivity: for any countable number of mutually exclusive events , we have

From the three axioms, we can derive many useful properties: - Proof: trivial. - Finite additivity: for any finite sequence of mutally exclusive events , Proof: straightforward of the additivity axiom with for all .

• The complement rule: . Proof: immediate from the finite additivity with .
• The numeric bound: . Proof: straightforward from the complement rule.
• Monotonicity: if , then and . Proof: from the finite additivity, since , we have . We also know that , so we have .
• The addition law: .
• The inclusion-exclusion principle:

Proof: We prove this by induction on . Base case: for :

trivially holds. for :

is exactly the same as the addition law.

Inductive case: Consider events sequence , then:

Then from the addition law:

From the distributive laws:

Let

Then:

Apply the inductive hypothesis to the events :

Computing Probabilities

Addition and Multiplication Principles

Addition principle: If there are types of methods to complete a task, with specific methods in the th type, then the total number of specific methods to complete this task is:

Multiplication principle: If there are steps to complete a task, with possible methods for the th step, then the total number of methods to complete this task is:

Definition: Permutation and Combination

Permutation: If elements are randomly selected without replacement from distinct elements () and placed in order, then the number of different permutation is

Combination: The number of combinations of elements randomly selected without replacement from distinct elements (), where the order does not matter, is given by

Example

Suppose a class of students has been allocated () concert tickets by the university. The teacher decides to distribute the tickets by drawing lots. The teacher prepares a hat containing slips of paper, with slips marked with 1 and the remaining slips marked with 0. The students take turns drawing slips from the hat, and those who draw a slip marked with a 1 will get a concert ticket. If you are one of the students in the class and you really want to get a ticket, would you choose to draw early or late?

Solution. The essential question is that, suppose you are the th to draw a slip, does the probability depend on ? Considering the ticket drawing as a random experiment, then each possible outcome of the drawing process can be represented by a -digit binary number, with digits being 1. Then the number of sample points is . w.l.o.g. we assume that . Then let denote the event that you get a ticket, so ,

Example

There are student in a class with student IDs . Now we prepare gifts numbered with each possible student IDs. Then everyone randomly select their gift from a big bag. What’s the probability that at least one student get a gift with his/her own ID?

Solution. Each permutation of is a sample point, representing the gift numbers corresponding to students with IDs . So . Let be the event that at least one student gets his/her own gift, then directly determining is not so straightforward. Let be the event that the student with ID get his/her own gift, then The events are very clear and their probability computation is simple: Put these together, we have:

Geometric Model of Probability

The classical model of probability assumes finite number of sample points and equal likelihood. Another model, called the geometric model of probability, is a natural extension of the classical model to an infinite number of sample points while maintaining equal likelihood.

Definition: Geometric Model of Probability

If a random experiment can be represented as randomly throwing a point onto a bounded region , where the point is equally likely to land at any position within the region, then the probability model of the experiment is called a geometric model of probability. Let be the event that the point lands at a subregion of , then the probability of event is computed as

Example

Bertrand’s paradox: Consider a circle with radius 1. What is the probability that a randomly draw chord of the circle is longer than the side of the inscribed equilateral triangle of the circle?

• Solution 1
• Solution 2
• Solution 3

Possible solution 1. Take a radius of the circle , and randomly choose a point on the radius, then draw a chord through orthgonal to . By elementary geometry, intersects the triangle at the midpoint of , say . Then the sample space is all points on . For the chord to be longer than the side of the triangle, must fall on , therefore, the probability is

Possible solution 2. Take a point on the circle, say , draw the tangent to the circle through . Then randomly choose another point on the circle, and draw a chord by connecting and that forms a random angle with the tangent. Then the sample space is , and for the chord to be longer than the side of the triangle, we need . Therefore, the probability is

Possible solution 3. Randomly choose a point within the circle and draw a chord with as the midpoint. For the chord to be longer than the side of the inscribed equilateral triangle, must fall within the inscribed circle of the triangle, whose radius is . Then the sample space is the original circle, the event of interest is the inscribed circle. Therefore, the probability is

Both of the three solutions are correct. The reason why the probabiliy varies across different solutions is that the hypothesis “the chord is randomly drawn” is not clearly defined. Different assumptions of equal likelihood lead to different sample spaces:

• Solution 1: equally likely chosen on a radius, so the sample space is the radius.
• Solution 2: equally likely chosen on the circle, so the sample space is the circle boundary.
• Solution 3: equally likely chosen within the circle, so the sample space is the whole circle area.

Therefore, when computing probabilities, it is crucial to clearly define the sample apce first.

Conditional Probability and Independence

Definition: Conditional Probability

Let and be two random events and . Then the conditional probability of event given that event occurs is defined as

The idea behind this definition is that if event has occured, the sample space becomes instead of .

Example

You are playing a poker game where you are dealt with 5 cards face down. If one of the cards that you are dealt lands face up, showing the Ace of spades, what is the probability of having a royal flush now?

Solution. Let be the event of having a royal flush. The probability of is straightforward: Let be the event that one of the 5 cards is the Ace of spades, then the number of sample points in is . is the event of a royal flush of spades. So the probability of given is

Multiplication Law

Let and be two random events and . Then .

Law of Total Probability

Let and be two random events, then (assume that if ) More generally, if are mutually exclusive random events, and , then we call a partition of the sample space , and

Just like how the addition law generalize to the inclusion-exclusion law, the multiplication law also has a generalized version, called the chain rule for random events. s

Chain Rule

assume that .

Bayes’ Theorem

Let be random events and is a partition of the sample space. Then for any event s.t. , we have is the probability of given that occurs, also termed the posterior possibility of . is called the prior probability or marginal probability of , which refers to the probability value in the absence of any other prior information.

A intuitive understanding of the Bayes’ rule and the law of total probability is:

• the law of total probability can be viewed as from cause to effect, since we are calculating the probability of the outcome event based on all possible causes.
• the conditional probability obtained by the Bayes’ rule can be viewed as the probability of the specifc cause led to the observed outcome . Simply put, we are reasoning from effect to put.

Definition: Independence

Let all denote random events. - and are said to be independent if . - are said to be mutually independent if for every subset of these events (, ), we have

Caution

Pairwise independence does not imply mutual independence.

Independence can sometimes rely on the occurrence of some random event. This is the concept of the conditional independence.

Definition: Conditional Independence

Let all denote random events. are said to be conditionally independent conditioned on if for every subset of the events, (, ), we have

Example

Draw three cards from a properly shuffled standard deck, with replacement and reshuffling. Let be the event that “card 1 and 2 have the same suit”, be the event that “card 2 and 3 have the same suit”, be the event that “card 1 and 3 have the same suit”. Are mutually independent?

Solution. It is trivial to show that . And are the same event “card 1, 2, 3 have the same suit”, so it is also trivial that . Therefore, we have but So the three events are pairwise independent, but not mutually independent.