We’ve released the latest version of the Factual PHP Driver: this includes enhanced features within our new Submit and Flag APIs, including clear and strict mode, as well as a number of new integration tests to make our collective lives easier during installation and testing. Diffs are now supported, and we also added a cheeky header to facilitate using the driver in a Wordpress Plugin. The changelog is here.
Processing data can be a real a mess!
Here at Factual we’ve felt the pain of managing data workflows for a very long time. Here are just a few of the issues:
a multitude of steps, with complicated dependencies
code and input can change frequently — it’s tiring and error-prone to figure out what needs to be re-built
inputs scattered all over (home directories, NFS, HDFS, etc.), tough to maintain, tough to sustain repeatability
Paul Butler, a self-described Data Hacker, recently published an article called “Make for Data Scientists”, which explored the challenges of managing data processing work. Paul went on to explain why GNU Make could be a viable tool for easing this pain. He also pointed out some limitations with Make, for example the assumption that all data is local.