Using curl and POST requests to automate data retrieval from a website

Problem: A site contains data which is of interest to us. The data is uniformly structured and available for examination by the general public (i.e. no login required) but is not automatically accessible, i.e. viewing it requires repetitive user interaction (such as setting a date range, clicking a ‘submit’ button, etc). In addition, usually the […]

Continue reading