Batch processing for big data
When it comes to handling large amounts of data, there is really only one way to reliably do it: batch processing. The concept of batch processing is simple. Instead of performing one large query and then parsing / formatting the data as a single process, you do it in batches, one small piece at a time. If you have ever attempted to query or export a large amount of data and had your server timeout, you’ll easily understand just how beneficial batch processing is.
