I am working on a new feature (EventExtraction Tool) to add to the main ExtractionEngine program.  To test the code I used the new tool and pulled the dates and filing types for GE from the database I built and compared it to the list of filings from GE’s SEC filing page.  The listing from my database was about 60 filings short when compared to the SEC filings list.

If you can imagine I started panicking because I couldn’t identify why the count differed so substantially  since this would clearly have implications regarding the claims I have made for the completeness of directEDGAR’s main filing repository.  After checking every possible reason for the difference due to problems with my coding I finally decided to compare the index list to what was in my database as well as to the actual filings.  That is where the problem turned out to be.

The quarterly SEC indexes are incomplete.   I have reached out to my contact at the SEC regarding the discrepancy and they are working on a fix.  The filings that seem to be missing from the indexes most often are those that have a filing date in the header file that is different from the actual date the final document was submitted to the EDGAR system.

The most common type of filings that seem to be missing are UPLOAD, CORRESPONDENCE and S-4 registration filings.  It looks to me that more than half of the CORRESPONDENCE and UPLOAD types are missing.   Less than 1/2 of one-percent of 10-K, 10-Q and 8-K filings are missing.

I intend to do a complete rebuild of directEDGAR to add more metadata.  I will use the new indexes for the next build.