The article explains how to export scraped data in Node.js using the built-in fs module for JSON files and a package for CSV files. It recommends formatting JSON output for readability and alerts users to potential issues with inconsistent object keys in CSV files, which can cause misalignment. By adhering to consistent object structures, users can efficiently save their data into formatted files for later analysis. Overall, the article emphasizes the importance of proper structure for successful data exporting.
Saving the data to a JSON file will require us to write our array data to a file, using the file system. Luckily, Node.js has the in-built filesystem (fs) module to take care of that.
The exportToJSON function leverages the fs module to asynchronously write your scraped data array to a scraped-data.json file. I recommend setting the third argument of JSON.stringify to 2 as shown above - this adds indentation to the output data, making it much easier to read.
That's a lot of code. Instead, we will use the package, which does all the heavy lifting for us. This function saves your scraped data to a file named anon-scraped.csv in the current directory, which you can open and process with any CSV-compatible software, including Excel and Google Sheets.
But there's a caveat: if your data contains varying object keys, that would lead to missing or misaligned values in the exported CSV file. Bear in mind that CSV files rely on a consistent object structure.
Collection
[
|
...
]