The full first pass on providing the raw data from the 13F filings is now available. There are approximately 63 million observations out of a total of about 68 million reported in all of the 13F HR filings. I merged some of the fields from the Cover Page (these are also in the 13FHEADERDATA) to save you the complication of doing so. I am not totally sure this was the right thing to do or not.

There is no way you can expect to pull everything in one pass. We are always balancing cost/power. I had about 2,800 CIKs and I wanted for testing to pull data for report dates from the second quarter of 2016 to the third quarter of 2020 and it took about 35 minutes to pull the over 4 million results.

It took every bit of memory of our Appstream instance to make this particular pull and to keep it active in the DB Browser. We will try to fiddle with some optimizing choices but there are always constraints. This data has been indexed by CIK and periodofreport_DATE.

Finally we will be pushing in more CUSIP-CIK mappings as we verify new matches.