[Part 4] Opportunity is missed by most people because it’s dressed in overalls and looks like work

“Opportunity is missed by most people because it's dressed in overalls and looks like work,”
– Thomas Edison.
Part 1
Part 2
Part 3

Testing 500 dual-core machines
Man, distributed computing is a debugging hog.Without getting too technical, we're finding the main roadblock in Project 88 (our crazy effort to predict stock prices) is the delay it takes to boot up a virtual computer — about 6 minutes.

So debugging the automated firing up of all these computers can only be done by…well…firing up the computers.

But once it's done, our back testing engine will be humming along at many orders of magnitude faster than anything you could buy at retail.

In fact, it doesn't really take much more to scale from 1000 cores to 10,000 other than more money.

According to my back of the napkin calculation, It would cost at least $30 million to create the network we're leasing from Microsoft's Azure cloud computing data center (it's in a secret location…even I have no idea where it is other than somewhere in California).

The way I view strategy design has completely changed now that computing power will no longer be a constraint.

It's like filling a swimming pool with Niagara Falls vs. a garden hose.

The difficulty of this project tells me that few would ever do it.

“Opportunity is missed by most people because it's dressed in overalls and looks like work,” said Thomas Edison.

My guess is that we're entering the territory Jim Simons pioneered a few decades ago.

He was a code breaker for the NSA, and they've been using distributed networks since there were distributed networks.

His hedge fund has made tens of billions in profits…who wouldn't want  piece of that action right?

Anyhoo, we increased our super-computing power to 2000 cores and are running what will be the most computationally involved back test of my 20-year career…today…as in right this second!

The initial results are interesting in that we probably won't be able to predict future prices for every stock every day of the week.

Some pattern are likely just random.

So we'll likely have a top 100 list instead of 3000.

It's still early though.

This is the pattern recognition stage of the project.

Next comes the post pattern stats, then the forecasting, more back testing, and finally…after 17 years…I get to see if my hypothesis is correct.

These are exciting times, and I'm glad to have you aboard to see the process to the finish.

Click here to view Part 5

Trade smart,