I have been reworking our code to parse sections of 10-K filings. As I was testing code and reviewing the documents the code is working on I became more and more annoyed at the way the html documents are constructed. If you have not noticed – many filings now consist of a very limited number of lines. For instance – here is a screen shot of the source of a 10-K:

All of the critical elements of the document that I want to work on are in line 11. While there are ways to inspect elements using some of the browser features – I find that is much easier and efficient for me to open the document in Notepad++ when I need to inspect the way elements are organized etc so I can write the appropriate XPATH to identify parts of the document to parse. But if I open that document up in Notepad++ I discover line number 11 is 2,217,885 characters wide. It really is brutal to review and browse this.

We are going to transition every htm file so that it is pretty_printed. Here is an image of part of the same file after it has been pretty_printed:

Overall this change should improve your ability to identify elements that you want to analyze in your code. All of the elements should be the same except for two changes – first the sourceline property will change. That should be a very useful feature as it adds another way to differentiate elements. Second, almost all of the elements will have a newline character (‘\n’) added to the tail. Those can be stripped out easily if you are working with text. To understand this compare the next two images. The first one is before the file was pretty_printed. The sourceline is 11 (recall that in the first image I showed that all of the document was on line 11). Also there is no content associated with the tail.

The next image is the same element after the document was pretty_printed. Now the sourceline is 844 and the tail has a new line appended to it (‘\n’). For those of you looking at these images real closely – the memory reference is different because these are different runs on the same document.

We will begin the transition this weekend, starting with the most recent filings and then work back in time. I want to wait until the weekend because I need to modify the code we use to capture and first process filings and I don’t want some unexpected issue (one of those things that I don’t know that I don’t know) does not delay or impede our daily update process.

Stay tuned as we have something more substantial and interesting coming up in the next couple of weeks.