When I set out to accomplish some home improvement project my wife and my amazing journeyman helper (my son) ask me how long I think it will take.  I set some number of hours, days or weeks and invariably it takes two to three times longer than planned.  I of-course, can keep my cheer during the process by my lovely bride sometimes gets annoyed because the disruption and mess are lasting longer than planned and my helper gets frustrated because after the initial rush of excitement he would probably prefer hanging out with other 13 year old boys (maybe girls too – not sure) rather than grinding away with his doting elderly dad.

The same happens when we plan a new release for directEDGAR.  We can easily see the start and the finish lines.  However, we never really understand how complex the terrain is until we are in the middle of the project.  I was hoping to release Version 4.04 by July and then it got pushed back until January and we are finally near the end.  It should be released in the next two weeks.

So what is new?

• Improved the speed of access to our artifacts by a factor of 50 or more.  If you try to download say 5,000 or so Executive Compensation tables or MDA Snippets it takes upwards of 30 minutes  to an hour.  Now the same task will take less than one minute!  To get the stats for this bogpost I did a test to extract MDA for 5,000 CIK-YEAR pairs – using the old interface it took 42 minutes, the new interface took 23 seconds!!
• We tweaked the Context Extraction to ignore the fields that were used in the search – fields are used only to focus the search and all fields are automatically included in every context or summary output.  This reduces the bulk of the Context Extraction so you don’t have to filter the csv file after the search is extracted to remove the irrelevant context lines.
• We added a feature so you can generate a CSV file that contains all the meta-data about directEDGAR files and artifacts in a specific folder.  This is for those cases when you access some artifact or copy files from the main repository to another location and you need details about the files and the filers.  The new feature allows you to select a directory, all the files in the directory will be listed and we also parse the file names to give you the CIK, RDATE etc of the artifacts.  If the files is not a directEDGAR artifact we only provide the file name.
• We made the DateFilter persistent.  This was probably a bad design choice in the beginning – because the date filter settings are ‘hidden’ after set we decided in the previous version to always reset the date filter to the default (no dates selected) after each search.  Some users have expressed a preference to have the filter persist across searches so we sorted out how to let you know the date filter is set for each search – you can clear it if needed.
• We improved the way you set values for some of the meta-data filters for your searches.  Now those can be set as you select the filter item from the list rather than having to force you to go back into the search box, find the open parenthesis, set your cursor and then type.
• We made some bug fixes including
  • Making sure the SmartBrowser opens in the right directory when you finish some process that causes the SmartBrowser to open.
  • When there is a problem with an input file (missing column or perhaps the file is still open and in-use) we let you know and give you the opportunity to close the file before moving forward rather than reporting an error that causes you to shut-down.
  • Making sure you can stop a search or other process that is running when you hit the Stop button.

The best thing about this release is the improvement to the artifact access.  When we started making director and executive compensation tables available directly we had imagined several hundred thousand artifacts and now we are over 2.5 million and adding more than one hundred thousand per month.  Our system was not designed for this.  We had a lot more to learn about how to manage the delivery than we imagined when we started.  However, there are some amazing folks working behind the scenes and they developed an infrastructure that will allow for a considerable amount of growth.  This is great because we are going to add new artifacts with the new infrastructure.  We have a large number of ideas we are waiting to implement until this release is pushed out to our customers.

Our system is designed to validate many characteristics of the director and executive compensation tables we extract and normalize.  One of the validation steps is to confirm that the reported total matches the sum of the components with an acceptable difference set at the absolute value of $10,000.

While it is easy to identify the reason for many of the exceptions (table reports 123.456 – value should be 123,456) others require more analysis and then ultimately some judgement.  I was working today and our system reported a $33,538 difference between the sum of the components and the reported total for CIK 1074902.  Here is the original table:

 

The reported total for Mr. Meilstrup in 2016 is $215,199, the sum of the components is $248,737, a difference of $33,538.

We reviewed the document and the table and wondered if the difference could be explained by the repetition of $45,793 in the NONEQUITY and NQDEFCOMP columns.  This seemed particularly likely since the amount reported for NONEQUITY in 2016 was more than three times the amount reported for the other officers of similar rank.  Thus we substituted the $12,255 amount reported in the 2016 column for the other Executive Vice Presidents and the difference between the components and the sum of the components dropped to $0.

Our final data push for this bank reflected the change we made