We loaded a new database on the platform today (FILING_TIMES). The database has all of the available submission times from the archive of the filings made available through the SEC Old Loads Archive. Unfortunately, the archive is/was incomplete as of 7/20/22 – the 2021 Q3 directory only includes filings from 7/1 though 7/9. I have already asked the SEC if these will be made available. We did some research into other sources available through EDGAR for compiled filing archives and the sources we have identified so far do not include the submission time. Another limitation is that there are no Old Load Archive entries prior to 1996.
What we did was merged the SEC INDEX Quarterly master.idx files with the submission line from the Old Loads Archive by accession number and used those results to create the database.
The primary key in the database is the concatenation of the CIK and the ACCESSION number separated with an underscore (CIK_ACCESSION). However, you can filter on CIK and query based on filing type. This database is huge and so I would advise caution in running queries that are not limited by CIK and date ranges. It is important to remember that more than one CIK can be associated with each filing. For ownership filings, the subject company CIK will be associated with the filing as well as the reporting person’s CIK. For companies that have subsidiaries that have reporting obligations each 10-K will be filed under the CIK of the parent and each of the subsidiaries.
The database has the following keys/fields:
- CIK_ACCESSION
- CIK
- ACCESSION
- FORM (SEC FORM NAME)
- FILING_DATE (as reported in the master.idx file)
- FILER_NAME
- SUBMISSION_TIME
The SUBMISSION_TIME is in YYYYMMDDHHMMSS format. If you wanted to process this as a date time object in Python (assume the date is in the variable SUBMISSION_TIME) date_obj = datetime.strptime(SUBMISSION_TIME, ‘%Y%m%d%H%M%S’).
According to the log there are 21,849,043 entries in the database. When I tried to retrieve all records on my test machine using our application it took approximately 20 minutes. While profiling this it was clear that the constraint is memory, the application had to make extensive use of the Windows Page (aka Swap) File. That is not very exciting. However, when I attempted a more realistic example, I had a list of 1,885 CIKs and set the search to retrieve all filings made by this set from 6/2/2007 to 10/3/2019 (note this date span was arbitrary). With this more realistic example the process took less than two minutes.
I saved that to a CSV file, but it is important to remember that Excel can only fully open files with a maximum 1,048,576 rows. This file has 2,193,942 rows.
As I am writing this I realize we might need to add some features to our Query panel to make your work easier. I did not think of including a NOT operator – and I wanted to repeat the query and exclude the ownership forms. An important point, you are not limited to the operators on the panel. If you are comfortable with SQLITE query syntax – you should be able to construct a query (limited of course by the available columns and their properties). So for example, while we did not include a NOT operator as a push button (there will be one of the next version) you can construct a query using NOT.
(FILING_DATE BETWEEN '2007-06-02' AND '2019-10-03') AND NOT (FORM IN ("3","4","5","3/A","4/A","5","A")) 
By excluding ownership forms the final file had only 903,906 rows.
We have more coming!