Let \(\mathbf{M}\) be the size of the range (i.e. for md5 it is \(2^{128}\) since md5 returns 128 [...]]]> One of the problems of hash functions is collisions, they cause some security vulnerabilities which won’t be discussed in this article.
In this article I would like to find some approximations concerning collisions.

Probability for a collision

Let \(\mathbf{M}\) be the size of the range (i.e. for md5 it is \(2^{128}\) since md5 returns 128 bits).
Let \(\mathbf{n}\) be the number of  random values in this range.
We shall calculate \(p\), the probability for at least one collision.

First, we calculate \(\overline{p}\), the probability for no collisions:
The first value can be any one from \(\mathbf{M}\), the second value can be any one from \(\mathbf{M}-1\) different values, and so on.
Therefore, \(\overline{p}=\frac{\mathbf{M}}{\mathbf{M}}\frac{\mathbf{M}-1}{\mathbf{M}}…\frac{\mathbf{M}-(\mathbf{n}-1)}{\mathbf{M}}=\prod_{k=0}^{\mathbf{n}-1}\frac{\mathbf{M}-k}{\mathbf{M}}=\prod_{k=0}^{\mathbf{n}-1}(1-\frac{k}{\mathbf{M}})\).

Since \(p=1-\overline{p}\) it is clear that:
\[(1) p=1-\prod_{k=0}^{\mathbf{n}-1}(1-\frac{k}{\mathbf{M}})\]

Recall that \(e^x=\sum_{k=0}^{\infty}\frac{x^k}{k!}\approx \frac{x^0}{0!}+\frac{x^1}{1!}=1+x\), substituting \(x=-\frac{k}{\mathbf{M}}\) gives \(1-\frac{k}{\mathbf{M}}\approx e^{-\frac{k}{\mathbf{M}}}\).
Now, from \((1)\):
\(p\approx 1-\prod_{k=0}^{\mathbf{n}-1}e^{-\frac{k}{\mathbf{M}}}=1-e^{\sum_{k=0}^{\mathbf{n}-1}{-\frac{k}{\mathbf{M}}}}=1-e^{-\frac{1}{\mathbf{M}} \sum_{k=0}^{\mathbf{n}-1}k}=1-e^{-\frac{\mathbf{n}(\mathbf{n}-1)}{2\mathbf{M}}}\)
Thus:
\[(2) p\approx 1-e^{-\frac{\mathbf{n}(\mathbf{n}-1)}{2\mathbf{M}}}\]

Under the condition \(1\ll \mathbf{n}^2\ll 2\mathbf{M}\) an even simpler formula emerges.
Again, \(e^x\approx 1+x\), substituting \(x=-\frac{\mathbf{n}^2}{2\mathbf{M}}\) gives \(1–\frac{\mathbf{n}^2}{2\mathbf{M}}\approx e^{-\frac{\mathbf{n}^2}{2\mathbf{M}}}\).
From \((2)\):
\(p\approx 1-e^{-\frac{\mathbf{n}(\mathbf{n}-1)}{2\mathbf{M}}}\approx 1-e^{-\frac{\mathbf{n}^2}{2\mathbf{M}}}\approx 1-(1-\frac{\mathbf{n}^2}{2\mathbf{M}})=\frac{\mathbf{n}^2}{2\mathbf{M}}\)
Therefore:
\[(3) p\approx \frac{\mathbf{n}^2}{2\mathbf{M}}\]

Amount of values for a probable collision

Let \(\mathbf{M}\) be the size of the range (i.e. for md5 it is \(2^{128}\) since md5 returns 128 bits).
Let \(\mathbf{p}\) be the desired probability for at least one collision.
We shall calculate \(n\), the number of values such that the probability for a collision is \(p\).

From \((2)\):
\(\mathbf{p}\approx 1-e^{-\frac{n(n-1)}{2\mathbf{M}}}
\\
1-\mathbf{p}\approx e^{-\frac{n(n-1)}{2\mathbf{M}}}
\\
\log(1-\mathbf{p})\approx-\frac{n(n-1)}{2\mathbf{M}}
\\
n(n-1)\approx-2\mathbf{M}\log(1-\mathbf{p})=2\mathbf{M}\log \frac{1}{1-\mathbf{p}}
\\
n^2-n-2\mathbf{M}\log \frac{1}{1-\mathbf{p}}\approx0
\\
n\approx\frac{1+\sqrt{1-8\mathbf{M}\log \frac{1}{1-\mathbf{p}}}}{2}=\frac{1}{2}+\sqrt{\frac{1}{4}+2\mathbf{M}\log \frac{1}{1-\mathbf{p}}}\)
Hence:
\[(4) n\approx \frac{1}{2}+\sqrt{\frac{1}{4}+2\mathbf{M}\log \frac{1}{1-\mathbf{p}}}\]

If \(\mathbf{M}\mathbf{p}\gg 1\) the constants are negligible:
\[(5) n\approx\sqrt{2\mathbf{M}\log \frac{1}{1-\mathbf{p}}}\]

If in addition \(\mathbf{p}\ll 1\), formula \((3)\) implies (can be demonstrated using \((5)\) too):
\[(6) n\approx \sqrt{2\mathbf{M}\mathbf{p}}\]

Examples

Birthday problem

The birthday problem (often referred as birthday paradox) states that if \(23\) people are randomly selected, the probability that two of them share their birthdays is higher than \(\frac{1}{2}\).
In this case \(\mathbf{M}=365\), \(\mathbf{n}=23\) and \(\mathbf{p}=\frac{1}{2}\).

Using formula\( (1)\):
\(p=1-\prod_{k=0}^{22}(1-\frac{k}{365})=0.507297\)

Luckily, formula \( (2)\) produces a similar result:
\(p\approx 1-e^{-\frac{23\times 22}{2\times 365}}=1-e^{-\frac{253}{365}}=0.500002\)

The inverse can be calculated using formula \( (4)\):
\(n\approx \frac{1}{2}+\sqrt{\frac{1}{4}+2\times 365\log \frac{1}{1-\frac{1}{2}}}=\frac{1}{2}+\sqrt{\frac{1}{4}+730\log 2}=22.9999\)

Formulas \( (3)\), \( (5)\) and \( (6)\) are inappropriate in this case (however their results are not too far from the actual results).

md5 hash function

In this case \(\mathbf{M}=2^{128}\) as I have mentioned in the beginning.

Using formula \((5)\) (\(\mathbf{M}\) is very large) we can find \(n\) given \(\mathbf{p}\):
\(\mathbf{p}=\frac{1}{1000}\): \(n\approx\sqrt{2\times 2^{128} \log \frac{1}{1-\frac{1}{1000}}}=8.25170\times 10^{17}\)
\(\mathbf{p}=\frac{1}{100}\): \(n\approx\sqrt{2\times 2^{128} \log \frac{1}{1-\frac{1}{100}}}=2.61532\times 10^{18}\)
\(\mathbf{p}=\frac{1}{10}\): \(n\approx\sqrt{2\times 2^{128} \log \frac{1}{1-\frac{1}{10}}}=8.4678\times 10^{18}\)
\(\mathbf{p}=\frac{1}{2}\): \(n\approx\sqrt{2\times 2^{128} \log \frac{1}{1-\frac{1}{2}}}=2.17194\times 10^{19}\)
\(\mathbf{p}=\frac{3}{4}\): \(n\approx\sqrt{2\times 2^{128} \log \frac{1}{1-\frac{3}{4}}}=3.07158\times 10^{19}\)
\(\mathbf{p}=\frac{99}{100}\): \(n\approx\sqrt{2\times 2^{128} \log \frac{1}{1-\frac{99}{100}}}=5.59832\times 10^{19}\)

For \(\mathbf{p}=\frac{1}{1000}, \frac{1}{100}, \frac{1}{10}\) formula \((6)\) can be used (higher values give bad results):
\(\mathbf{p}=\frac{1}{1000}\): \(n\approx\sqrt{2\times 2^{128} \frac{1}{1000}}=8.24963\times 10^{17}\)
\(\mathbf{p}=\frac{1}{100}\): \(n\approx\sqrt{2\times 2^{128} \frac{1}{100}}=2.60876\times 10^{18}\)
\(\mathbf{p}=\frac{1}{10}\): \(n\approx\sqrt{2\times 2^{128} \frac{1}{10}}=8.24963\times 10^{18}\)
These results are consistent with the previous results.

Summary

We have obtained some formulas that relate \(\mathbf{M}\), \(\mathbf{n}\) and \(\mathbf{p}\):
\[(1) p=1-\prod_{k=0}^{\mathbf{n}-1}(1-\frac{k}{\mathbf{M}})
\\
(2) p\approx 1-e^{-\frac{\mathbf{n}(\mathbf{n}-1)}{2\mathbf{M}}}
\\
(3) p\approx \frac{\mathbf{n}^2}{2\mathbf{M}} (1\ll \mathbf{n}^2\ll 2\mathbf{M})
\\
(4) n\approx \frac{1}{2}+\sqrt{\frac{1}{4}+2\mathbf{M}\log \frac{1}{1-\mathbf{p}}}
\\
(5) n\approx\sqrt{2\mathbf{M}\log \frac{1}{1-\mathbf{p}}} (\mathbf{M}\mathbf{p}\gg 1)
\\
(6) n\approx \sqrt{2\mathbf{M}\mathbf{p}} (\mathbf{Mp}\gg 1, \mathbf{p}\ll 1)\]

We need a data-structure that can contain integers.
It must have 2 methods which are add [...]]]> After discussing with some friends on StartupSeeds, I came up with a generalized random sub array algorithm.
This algorithm can be implemented with different data-structures, and it’s complexity is \(O(n)\) or \(O(k\log\space k)\) for 2 specific implementations.

We need a data-structure that can contain integers.
It must have 2 methods which are add (adds an integer) and contains (whether an integer is in or not).
We also need to be able to iterate it, I will use foreach loop for that.

The Algorithm

Let \(A\) be the original array, with elements of type \(T\).
Let \(k\) be the number of elements we want in the sub array.
Let \(n=length(A)\).
Let \(D\) be the data-structure.

1. Let \(B=T[k]\) (an array of type \(T\) with \(k\) elements)
2. Let \(flag=(k\le n / 2)\)
3. Let \(stop=(flag ? k : n-k)\)
4. \(for(i=0;i\lt stop;i=i+1)\)
   1. \(while(true)\)
      1. Let \(key=random(0,n-1)\)
      2. \(if(!D.contains(key))\)
         1. \(D.add(key)\)
         2. \(break\)
5. \(if(flag)\)
   1. Let \(p=0\)
   2. \(foreach(key\space in\space D)\)
      1. \(B[p]=A[key]\)
      2. \(p=p+1\)
6. \(else\)
   1. \(keys=bool[n]\)
   2. \(foreach(key\space in\space D)\)
      1. \(keys[key]=true\)
   3. Let \(p=0\)
   4. \(for(i=0;i\lt k;i=i+1)\)
      1. \(while(keys[p])\)
         1. \(p=p+1\)
      2. \(B[i]=array[p]\)
      3. \(p=p+1\)
7. \(return\space B\)

Complexity Analysis

I will calculate each stage’s complexity separately, and then sum them up.
But first:
Let \(T_c(j)\) be the complexity of \(D.contains\) when there are \(j\) elements in it.
Let \(T_a(j)\) be the complexity of \(D.add\) when there are \(j\) elements in it.
Let \(T_i(j)\) be the complexity of iterating the data-structure’s elements when there are \(j\) elements in it.

Stage 1 is \(O(k)\) or \(O(1)\) depending on the language’s implementation for array declarations, we will use \(O(k)\).

Stage 2 and 3 are clearly \(O(1)\).

Stage 4 is more complex to calculate and we can only calculate it for the average case.
In each of the iterations in the for-loop we are looking for a number between \(0\) and \(n-1\) that is not already in \(D\).
Because there are already \(i\) elements in the data-structure, the distribution of the number of while-loop iterations is a geometric distribution with probability \(\frac{n-i}{n}\), therefore the mean number of iterations is \(\frac{n}{n-i}\).
So this stage’s complexity is:
\[\displaylines{O(\sum_{i=0}^{min(k,n-k)-1}(T_c(i)\frac{n}{n-i}+T_a(i)))
\\
=O(\sum_{i=0}^{min(k,n-k)-1}(T_c(i)\frac{n}{n/2}+T_a(i)))
\\
=O(\sum_{i=0}^{min(k,n-k)-1}(T_c(i)+T_a(i)))}\]

Stage 5 is simply \(O(T_i(k))\).

Stage 6 is also quite simple.
There is an array deceleration (\(O(n)\) as mentioned before), iteration over the data-structure \(O(T_i(n-k))\) and an iteration on the arrays (\(O(n)\).
Summing up to \(O(T_i(n-k)+n)\).
But we know that \(k\ge n/2\), and therefore it is also \(O(T_i(n-k)+k)\).

Bear in mind that only one of stages 5 and 6 occurs, hence there total complexity of these stages is \(O(T_i(min(k,n-k))+k)\).

Stage 7 is also \(O(1)\).

Summing stages 1 to 7 gives us:
\[O(\sum_{i=0}^{min(k,n-k)-1}(T_c(i)+T_a(i))+T_i(min(k,n-k))+k)\]

Implementation #1 – Boolean Array

This implementation was introduced in my previous article, Generating Random Sub Array.
In this implementation the data-structure acts the same as the array \(keys\) that is used in stage 6.1.

In this case \(T_c(j)=1\), \(T_a(j)=1\) and \(T_i(j)=n\).
With that in mind the algorithms complexity can be found easily: \(O(\sum_{i=0}^{min(k,n-k)-1}(1+1)+n+k)=O(n)\)

Implementation #2 – Self-Balancing Binary Search Tree

In this implementation our data-structure is a binary search tree.
This implementation is much better than the one mentioned above when \(k\ll n\).

In this case \(T_c(j)=\log\space j\), \(T_a(j)=\log\space j\) and \(T_i(j)=j\).
We can again easily find the total time complexity.
For simplicity I will assume that \(k\le n/2\).
\(O(\sum_{i=0}^{k-1}(\log\space j+\log\space j)+k+k)=O(k\log\space k)\)

I came up with an algorithm with average [...]]]> Today I came across an algorithmic problem.
I wanted to choose a random sub array of size \(k\) from an array of size \(n\), while maintaining its order.
For example, if we have the array \(\{0,1,2,3,4,5,6,7,8,9\}\) a possible result for \(k=4\) is the array \(\{1,4,7,9\}\).

I came up with an algorithm with average case time complexity \(O(n)\).

The Algorithm

Let \(A\) be the original array, with elements of type \(T\).
Let \(k\) be the number of elements we want in the sub array.
Let \(n=length(A)\).

1. Let \(B=T[k]\) (an array of type \(T\) with \(k\) elements)
2. Let \(keys=bool[n]\) (an array with \(n\) boolean elements)
3. Let \(flag=(k\le n / 2)\)
4. Let \(stop=(flag ? k : n-k)\)
5. \(for(i=0;i\lt stop;i=i+1)\)
   1. \(while(true)\)
      1. Let \(key=random(0,n-1)\)
      2. \(if(keys[key]=false)\)
         1. \(keys[key]=true\)
         2. \(break\)
6. Let \(p=0\)
7. \(for(i=0;i\lt k;i=i+1)\)
   1. \(while(keys[p] \oplus flag)\) (where \(\oplus\) is xor)
      1. \(p=p+1\)
   2. \(B[i]=array[p]\)
   3. \(p=p+1\)
8. \(return\space B\)

Average Case Time Analysis

Stages 1 and 2 depends on the language’s implementation for array declaration.
However they are always \(O(n)\).

Stages 3, 4 and 6 are clearly \(O(1)\).

Let’s take a look at stage 7.
Since \(p\) can never be bigger than \(n\), and \(i\) can never be bigger than \(k\), and each iteration is \(O(1)\), these stages are \(O(n+k)=O(n)\).

Now, let’s consider stage 5.
Within each step of the loop we are trying to find a number between \(0\) and \(n-1\) which wasn’t picked earlier.
The distribution of the number of guesses it takes to find such a number, is a geometric distribution with probability \(\frac{n-i}{n}\), hence the mean is \(\frac{n}{n-i}\).
We know that \(i\) goes from \(0\) to \(min(k,n-k)-1\) which is less than \(n/2\).
Each step in the while loop is \(O(1)\).
Therefore the worst case complexity, on average, is \(O(\sum_{i=0}^{n/2-1}\frac{n}{n-i})=O(\sum_{i=0}^{n/2-1}\frac{n}{n/2})=O((n/2)\frac{n}{n/2})=O(n)\).

Summing all of the stages together, we can see that the algorithm has an average case time complexity of \(O(n)\).

C# implementation

I wrote a simple implementation of the algorithm in C#:

public static T[] RandomSubArray<T>(T[] array, int k)
{
	int n = array.Length;

	T[] ret = new T[k];
	bool[] keys = new bool[n];
	Random rand = new Random();

	bool flag = k <= n / 2;

	for (int i = 0, stop = flag ? k : n - k; i < stop; i++)
	{
		while (true)
		{
			int key = rand.Next(0, n);
			if (!keys[key])
			{
				keys[key] = true;
				break;
			}
		}
	}

	int p = 0;
	for (int i = 0; i < k; i++)
	{
		while (keys[p] ^ flag)
		{
			p++;
		}
		ret[i] = array[p++];
	}

	return ret;
}

A phrase based password is a password assembled from several easy to remember and spell words, delimited by spaces.
As a result, such passwords are very [...]]]> Yesterday I have published an article about password entropy.
Today I would like to discuss the entropy of a phrase based password.

Phrase Based Password

A phrase based password is a password assembled from several easy to remember and spell words, delimited by spaces.
As a result, such passwords are very easily remembered.

To produce such a password, one must have a dictionary of words.
Each time a user asks for a password, the system randomly chooses a few words to generate the password.

The Entropy of a Phrase Based Password

Let \(N\) be the size of our dictionary, and let \(L\) be the number of words in the password, therefore there are \(T = {N \choose L}L!\) different possible passwords.
Assuming \(N \gg L\) we can approximate this number by \(T \approx N^L\).
As we know the entropy \(H\) is given by \(H = \log_2{T}\), thus \(H \approx \log_2{N^L} = L \log_2{N}\), which is identical to the entropy of a password of length \(L\) under an alphabet with \(N\) different symbols.

Let’s assume that \(N = 10,000\) (i.e. we have 10,000 unique words in our dictionary), and \(L = 5\), the entropy of a password under these conditions is \(H \approx 5 \log_2{10,000} \approx 66.44 \text{bits}\).
Unfortunately when \(N = 3,000\) and \(L = 4\) the entropy is much lower: \(H \approx 4 \log_2{3,000} \approx 46.2 \text{bits}\)

Conclusions

Phrase base passwords are easy to remember; hence they are great in terms of ease of use.
On the other hand, in order for this method to be reliable, the dictionary has to big quite large, and each password must contain at least 4 or 5 words.

The method itself is very simple, you give an initial guess of the root, \(x_0\), and calculate further values using the following equation: \(x_{n+1}=x_n-\frac{f’(x_n)}{f(x_n)}\).
Generally, each term in the sequence is more accurate in comparison to the previous terms.

PDO is an abstraction layer for database connections in PHP, and it became increasingly popular in the past few years.

First of all, my algorithm can only find roots between a specific range (for example \(0\) to \(1\)), thus we need to decrease [...]]]> A few days ago I thought of an algorithm to calculate square roots. In this article I would like to introduce my algorithm.
I should mention that this algorithm probably exists.

First of all, my algorithm can only find roots between a specific range (for example \(0\) to \(1\)), thus we need to decrease the number whose root we are looking for to our range.
To do so we use a simple math identity: \(\sqrt{p^2x}=p\sqrt{x}\).
All we have to do is pick a number \(p\), divide \(x\) by \(p^2\) until it’s square root is in our range, and remember the number of times we repeated it – \(q\). When we have the result we need to multiply it by \(p^q\).

For example, if the range is \(0\) to \(1\), \(x=289\) and \(p=2\), we have to divide \(289\) by \(p^2=4\) \(q=5\) times to get \(0.2822265625\). Then, we use the algorithm I will explain later to find the root which is \(0.53125\). Now we can return \(0.53125*2^5=17\).
And indeed \(\sqrt{289}=17\).

The Algorithm

The algorithm is very simple and uses the same concept of binary search; it works by minimizing the possible range of the square root.
I use the algorithm in the range \(0\) to \(1\) but it can be used with any range.

We know that \(x\) is between \(0\) and \(1\), therefore its square root’s lower bound is \(a=0\) and upper bound is \(b=1\).
The next step is calculating the average of the bounds \(t=(a+b)/2\).
If \(t^2=x\) we return \(t\), if \(t^2<x\) all of the numbers between \(a\) and \(t\) are not the square root, hence \(a=t\). Similarly, if \(t^2>x\) our upper bound, \(b=t\).
We can repeat this step as many times as we want. Each iteration doubles the precision.
If we didn’t find the specific square root when we finish, we should return \((a+b)/2\), as it is the closest we can get to the actual square root.

The Code

I will demonstrate the whole algorithm in C#. In the following implementation \(p\) and the number of iterations are parameters.

public double Sqrt(double x, int p, int iterations)
{
	if (x < 0)
		throw new Exception("Can not sqrt a negative number");

	long multiplier = 1;
	int p2 = p * p;
	while (x > 1)
	{
		multiplier *= p;
		x /= p2;
	}

	if (x == 1 || x == 0)
		return multiplier * x;

	double a = 0;
	double b = 1;

	for (int i = 0; i < iterations; i++)
	{
		double t = (a + b) / 2;
		if (t * t == x)
			return multiplier * t;
		else if (t * t < x)
			a = t;
		else
			b = t;
	}

	return multiplier * (a + b) / 2;
}

Complexity

Rather than calculating the complexity of \(k\) iterations, I will calculate the complexity for precision of \(1\) to \(n\).

The first part consists of dividing \(x\) by \(p^2\) and multiplying the multiplier by \(p\), which is \(O(1)\) for each iteration.
\(x\) is divided \(\log_{p^2} x\) times, hence this part’s complexity is \(O(\log_{p^2} x)\).
Multiplying the result by the multiplier decreases the precision so we will take it in to consideration later. Finding the value of multiplier is simple: \(p^{\log_{p^2} x}=p^{(\log_p x)/(\log_p p^2)}=\sqrt{p^{\log_p x}}=\sqrt{x}\)

In the main part each iteration has a complexity of \(O(1)\) and it doubles the precision, as I explained before.
Let \(i\) be the minimal number of iterations to get a precision of \(1\) to \(n\).
That means that \(1/n=\sqrt{x}/2^i \Rightarrow 2^i=n \sqrt{x} \Rightarrow i=\log_2 (n \sqrt{x})\).
Therefore the total complexity of the main part is \(O(\log_2 (n \sqrt{x}))\).

Summing these two, the total complexity is
\(O(\log_{p^2} x + \log_2 (n \sqrt{x}))
\\=O((\log_p x)/(\log_p p^2) + \log_2 n + \log_2 \sqrt{x})
\\=O(0.5 \log_p x + \log_2 n + 0.5 \log_2 x)
\\=O(\log_2 x + \log_p x + \log_2 n)\)

We know that \(p \ge 2 \Rightarrow \log_p x \le \log_2 x\), therefore the worst-case complexity is \(O(\log_2 x + \log_2 x + \log_2 n)=O(\log_2 x + \log_2 n)=O(\log_2 nx)\).

Conclusions

I don’t know many square root algorithms which means that I don’t know other algorithm’s complexity. That said my algorithm is still very efficient.
More over, this method seems pretty intuitive to me, it is easy to understand and easy to implement.
It is also very easy to implement an algorithm that calculates square roots with a constant maximum error.

I hope someone will find this article useful.

Update 24/05/2011:
I read about Newton’s method, which is a method used to find the roots of functions. I found out that it is far more easy to implement and much more efficient in comparison to my method.