About Us
Optimus Mine enables people to gather massive amounts of data, so they can create new data driven applications, extract market insights, and acquire the information they need to grow their business.
Our Story
Optimus Mine was founded by Vincent Lee (data evangelist) and Adam Bouqdib (open source veteran) who are both data enthusiasts in search of a better way to acquire and analyse big data.
It started with a web scraping project where Vincent’s scripts were crashing and he was dealing with an operations nightmare. He had no idea on how he was going to fetch the data from 50 million web pages in just a few weeks. Somehow Vincent had to make this work, as his clients were depending on it.
September 2016, Vincent contacted Adam about this problem as they had worked together on many projects before. The prototype idea was to build a simple crawler that could fetch raw HTML from 50 million web pages in just a few weeks.
Being the perfectionist and open source obsessed programmer Adam is, he could not stop there…
Streaming parsers were added to the scripts so a number of different data formats could be consumed, uploaders were created so Vincent could analyse the data directly in Google BigQuery.
The job was completed, and from then on Optimus Mine was rolled out into production for many client projects.
Today you can use Optimus Mine on Windows, Apple, or Linux systems, and you can start building data driven businesses.