Creating a Local Data Lake Companion code and slides to my talk about whether you need Hadoop. A lot of processing can be done in-memory on your laptop, if you have a reasonably modern laptop. For example, you can run MapReduce with PyPy.