Unit Testing Browse and Doc It

I thought I would write something different in the article. The last month or so has been devoted to updating my Browse and Doc It IDE add-in, more specifically, updating the Object Pascal parser to understand Attributes, Generics and Anonymous Methods. The work is almost complete with 3 tests still failing: 2 new tests for anonymous methods and 1 existing test for creating comments.

I thought it might be interesting for those of you out there who have tried unit testing but found the concept difficult to implement because your code doesn't fit into the nice one simple test per function / property model. So here I'll explain how I've solved some of my problems which may provide ideas for all of you.

I'll say this now before starting – I have broken one of the rules for unit testing as described by Kent Beck, i.e. I do more than one test per test function. I've done this for a number of reasons: I was writing the same template code in multiple test methods which screamed refactoring at me; I have limited time to do my development (not the day job) so needed to maximise my output and finally I feel its an unnecessary restriction so long as you provide feedback in your testing code with additional information, for instance, I currently use DUnit and thus the code returns extended information in the third comment field of the checks to tell where the tests fails. The one down side of this is that the code will fail at the first test so once you've fix the code another test may fail.

The one thing I will say is I agree with the concept of writing your tests first before you implement your code (you will have to write the stubs otherwise it will not compile). This way you have a target to code to. Write the code and just press F9 to see if it works.

The first thing I did for my tests in Browse and Doc It was to create a few class helper functions to allow me to get information from my parser classes (they are all derived from a base class). An explanation of the parser structure is needed to understand what I've done in the following code.

The Browse and Doc It code is based on a class hierarchy which has a base class called TElementContainer from which every object should be derived. This base class provide the ability to create a data hierarchy by adding classes based on TElementContainer to other classes derived from the base class. The module that is shown in the module explorer is just a specific case of the base class with each node in the tree being a class based on the base class added to the module class to create the structure.

Below is the definiton of the class helper. See the implementation below for details on how and why there were implemented. These allow repeatative actions to be simplified but not coded into the main code where it's not necessary.

Type
  TElementContainerHelper = Class Helper for TElementContainer
    Function FirstError : String;
    Function FirstWarning : String;
    Function FirstHint : String;
    Function DocConflict(iConflict : Integer) : String;
    Procedure DeleteDocumentConflicts;
    Function HeadingCount(strHeading : String) : Integer;
  End;

This method deletes all the documentation conflicts which are added when parsing a module (the types of conflicts to be logged are configurable). This was needed to test comment conditions with and without conflicts.

procedure TElementContainerHelper.DeleteDocumentConflicts;

Var
  i : Integer;

begin
  For i := ElementCount DownTo 1 Do
    If Elements[i].AsString(True, False) = strDocumentationConflicts Then
      DeleteElement(i);
end;

This method allows me to get the text of a specific documentation conflict so that I can tell that I'm getting the right conflict for the circumstances that I'm testing for.

function TElementContainerHelper.DocConflict(iConflict : Integer): String;

Var
  E : TElementContainer;

begin
  Result := '(No Documentation Conflicts)';
  E := FindElement(strDocumentationConflicts);
  If (E <> Nil) And (E.ElementCount > 0) Then
    Begin
      E := E.Elements[1];
      If E.ElementCount >= iConflict Then
        Begin
          E := E.Elements[iConflict];
          Result := Format('%d) %s', [iConflict, E.AsString(True, False)]);
        End;
    End;
end;

This method gets the first error from the module if one exists. Note: All modules are expected to provide four nodes within the module: one for errors, another for warnings, another for hints and a final one for documentation conflicts.

function TElementContainerHelper.FirstError: String;

Var
  E : TElementContainer;

begin
  Result := '';
  E := FindElement(strErrors);
  If E <> Nil Then
    Result := StringReplace(Format('  [%s]', [E.Elements[1].AsString(True, False)]),
      #13#10, '(line-end)', [rfReplaceAll]);
end;

This method gets the first hint from the module if one exists.

function TElementContainerHelper.FirstHint: String;

Var
  E : TElementContainer;

begin
  Result := '';
  E := FindElement(strHints);
  If E <> Nil Then
    Result := StringReplace(Format('  [%s]', [E.Elements[1].AsString(True, False)]),
      #13#10, '(line-end)', [rfReplaceAll]);
end;

This method gets the first warning from the module if one exists.

function TElementContainerHelper.FirstWarning: String;

Var
  E : TElementContainer;

begin
  Result := '';
  E := FindElement(strWarnings);
  If E <> Nil Then
    Result := StringReplace(Format('  [%s]', [E.Elements[1].AsString(True, False)]),
      #13#10, '(line-end)', [rfReplaceAll]);
end;

This last method of the helper returns the number of headings (first level under the root module node).

function TElementContainerHelper.HeadingCount(strHeading : String): Integer;

var
  E: TElementContainer;

begin
  Result := 0;
  E := FindElement(strHeading);
  If E <> Nil Then
    Result := E.ElementCount;
end;</pre>

  <pre>TBaseLanguageModuleHelper = Class Helper For TBaseLanguageModule
    Function CurrentToken : TTokenInfo;
  End;</pre>

  <pre>function TBaseLanguageModuleHelper.CurrentToken: TTokenInfo;
begin
  Result := Token;
end;

The next thing I did was create a descendant class from the main TTestCase in order to add some new CheckEquals functions for types which are specific to Browse and Doc It. I've also overridden the CheckEquals for string comparisons to provide feedback on where in the string the mismatch occurs (helps with long strings). Finally I've added my generic grammar checking routine that allows me to check grammar for errors, warnings and hints plus check the structural output is as expected.

Type
  TExtendedTestCase = Class(TTestCase)
  Strict Private
  Public
    Procedure CheckEquals(ttExpected, ttActual : TBADITokenType; strMsg : String = ''); Overload;
    Procedure CheckEquals(iiExpected, iiActual : TBADIImageIndex; strMsg : String = ''); Overload;
    Procedure CheckEquals(iiExpected : TBADIImageIndex; iActual : Integer; strMsg : String = ''); Overload;
    Procedure CheckEquals(scExpected, scActual : TScope; strMsg : String = ''); Overload;
    Procedure CheckEquals(trExpected, trActual : TTokenReference; strMsg : String = ''); Overload;
    Procedure CheckEquals(ctExpected, ctActual : TCommentType; strMsg : String = ''); Overload;
    Procedure CheckEquals(strExpected, strActual : String; strMsg : String = ''); Overload; Override;
    Procedure TestGrammarForErrors(Parser : TBaseLanguageModuleClass;
      strTemplate : String; strInterface, strImplementation : String;
      TestTypes : TTestTypes; Const strCheckValues : Array Of String);
  Published
  End;

This first CheckEquals method allows me to check the enumerate which identifies the different types of tokens processed by the parsers. These tokens types are things like Identifiers, Reserved Words, Symbols, Strings, etc.

procedure TExtendedTestCase.CheckEquals(ttExpected, ttActual: TBADITokenType;
  strMsg: String = '');

begin
  FCheckCalled := True;
  If CompareText(strTokenType[ttExpected], strTokenType[ttActual]) <> 0 Then
    FailNotEquals(strTokenType[ttExpected], strTokenType[ttActual], strMsg,
      ReturnAddress);
end;

This CheckEquals method allows me to compare image indexes. Each type of object in the browser is associated with an image and this allows me to check that the correct images are associated with various items.

procedure TExtendedTestCase.CheckEquals(iiExpected, iiActual: TBADIImageIndex;
  strMsg: String = '');

begin
  FCheckCalled := True;
  If iiExpected <> iiActual Then
    FailNotEquals(BADIImageList[iiExpected].FResourcename,
      BADIImageList[iiActual].FResourcename,
      strMsg, ReturnAddress);
end;

This CheckEquals method allows me to check the scope associated with each element's output by the parsers (the usual, Private, Public, Local, etc.).

procedure TExtendedTestCase.CheckEquals(scExpected, scActual: TScope;
  strMsg: String);

Const
  strScopes : Array[Low(TScope)..High(TScope)] Of String = (
    'scNone', 'scGlobal', 'scLocal', 'scPrivate', 'scProtected', 'scPublic',
    'scPublished', 'scFriend');

begin
  FCheckCalled := True;
  If CompareText(strScopes[scExpected], strScopes[scActual]) <> 0 Then
    FailNotEquals(strScopes[scExpected], strScopes[scActual], strMsg,
      ReturnAddress);
end;

The next function allow for testing a token reference. For local and private variables I try to determines whether the token has been used in the code and mark them appropriately. I don't do this for any other scope as this would require parsing all the interface sections of the files included in the Uses clauses (something for the future perhaps).

procedure TExtendedTestCase.CheckEquals(trExpected, trActual: TTokenReference;
  strMsg: String);

Const
  strTokenReference : Array[Low(TTokenReference)..High(TTokenReference)] Of String = (
    'trUnknown', 'trUnresolved', 'trResolved');

begin
  FCheckCalled := True;
  If CompareText(strTokenReference[trExpected], strTokenReference[trActual]) <> 0 Then
    FailNotEquals(strTokenReference[trExpected], strTokenReference[trActual], strMsg,
      ReturnAddress);
end;

This method allows me to check the type of comments to be used under various circumstances (i.e. different languages).

procedure TExtendedTestCase.CheckEquals(ctExpected, ctActual: TCommentType;
  strMsg: String);

Const
  strCommentTypes : Array[Low(TCommentType)..High(TCommentType)] Of String = (
    'ctNone', 'ctPascalBlock', 'ctPascalBrace', 'ctCPPBlock', 'ctCPPLine',
    'ctVBLine', 'ctXML');

begin
  FCheckCalled := True;
  If CompareText(strCommentTypes[ctExpected], strCommentTypes[ctActual]) <> 0 Then
    FailNotEquals(strCommentTypes[ctExpected], strCommentTypes[ctActual], strMsg,
      ReturnAddress);
end;

This next method is an overridden check for strings. This check outputs infromation about where in the string the difference occurs (useful for long strings where it can be difficult to see the differences).

Procedure TExtendedTestCase.CheckEquals(strExpected, strActual: String;
  strMsg: String);

Var
  i : Integer;
  iPosition : Integer;

Begin
  FCheckCalled := True;
  If CompareText(strExpected, strActual) <> 0 Then
    Begin
      iPosition := 0;
      For i := 1 To Length(strExpected) Do
        If Length(strActual) >= i Then
           If strActual[i] <> strExpected[i] Then
             Begin
               iPosition := i;
               Break;
             End;
      If iPosition = 0 Then
        strMsg := strMsg + ' {Actual too small}'
      Else
        strMsg := strMsg + Format( ' [[Difference @ character %d: %s]]',
          [iPosition, Copy(strActual, 1, iPosition)]);
      FailNotEquals(strExpected, strActual, strMsg, ReturnAddress);
    End;
End;

Finally the next method is the multi-purpose routine for checking the parsers for errors, warning and hints but also the structural output of the module.

This method has the following parameters:

• Parser: This is a class reference to the base language module so that different languages base on that class can be passed to the testing code (they all have the same constructor) and be created by this method;
• strTemplate: This is a template for the below 2 parameters to be inserted into;
• strInterface: This is primarily the interface code for a unit template;
• strImplementation: This is primarily the implemntation code for a unit template;
• TestTypes: This is a set of test enumerates that allows the check to test for Errors, Warnings and Hints – they should not exist;
• strCheckValues: This is an array of string values each of which describe the strucure of an element of the module output in a hierarchical nature.

If Errors, Warning or Hints are required to be tested for then the code will check to ensure the count of Errors, Warnings or Hints is zero else it will fail and return a comment message displaying the first Error, Warning or Hint.

The rest of the code progressively searches the module structure for the given elements in the last parameter checking the output at each level of the hierarchy along with the elements scope.

Procedure TExtendedTestCase.TestGrammarForErrors(Parser : TBaseLanguageModuleClass;
  strTemplate : String; strInterface, strImplementation: String; TestTypes : TTestTypes;
  Const strCheckValues : Array Of String);

Const
  cDelimiter : Char = '\';

  Function GetElements(Element, ParentElement  : TElementContainer) : String;

  Var
    i : Integer;

  Begin
    Result := '';
    If Element = Nil Then
      Begin
        For i := 1 To ParentElement.ElementCount Do
          Begin
            If Result <> '' Then
              Result := Result + ', ';
            Result := Result + '[' + ParentElement.Elements[i].Identifier + ']';
          End;
      End;
  End;

  Function SearchForElement(Element : TElementContainer; strValue : String) : TElementContainer;

  Begin
    Result := Element.FindElement(strValue);
    If Result = Nil Then
      Result := Element.FindElement(strValue, ftIdentifier);
  End;

Var
  P: TBaseLanguageModule;
  T, U : TElementContainer;
  strValue: String;
  iCheck: Integer;
  strCheckValue: String;
  i : Integer;
  strKey : String;
  strValueScope : String;

Begin
  P := Parser.CreateParser(Format(strTemplate, [strInterface,
    strImplementation]), 'TestSource.pas', False, [moParse]);
  Try
    If ttHints In TestTypes Then
      CheckEquals(0, P.HeadingCount(strHints), 'HINTS: ' + P.FirstHint);
    If ttWarnings In TestTypes Then
      CheckEquals(0, P.HeadingCount(strWarnings), 'WARNINGS: ' + P.FirstWarning);
    If ttErrors In TestTypes Then
      CheckEquals(0, P.HeadingCount(strErrors), 'ERRORS: ' + P.FirstError);
    For iCheck := Low(strCheckValues) to High(strCheckValues) Do
      If strCheckValues[iCheck] <> '' Then
        Begin
         strCheckValue := strCheckValues[iCheck];
          T := P;
          While (Pos(cDelimiter, strCheckValue) > 0) And (Pos(cDelimiter, strCheckValue) < Pos('|', strCheckValue)) Do
            Begin
              strValue := Copy(strCheckValue, 1, Pos(cDelimiter, strCheckValue) - 1);
              Delete(strCheckValue, 1, Pos(cDelimiter, strCheckValue));
              U := SearchForElement(T, strValue);
              Check(U <> Nil, Format('%d.2) %s not found (found %s): %s', [Succ(iCheck), strValue, GetElements(U, T), strCheckValues[iCheck]]));
              T := U;
            End;
          Check(T.ElementCount > 0, Format('%d.3) Element Count: ', [Succ(iCheck), strCheckValues[iCheck]]));
          i := Pos('|', strCheckValue);
          Check(i > 0, Format('%d.4) Cannot find KEY to search for! ', [Succ(iCheck), strCheckValues[iCheck]]));
          strKey := Copy(strCheckvalue, 1, i - 1);
          Delete(strCheckValue, 1, i);
          i := Pos('|', strCheckValue);
          Check(i > 0, Format('%d.5) Cannot get scope: %s', [Succ(iCheck), strCheckvalues[iCheck]]));
          strValueScope := Copy(strCheckValue, i + 1, Length(strCheckValue) - i);
          Delete(strCheckValue, i, Length(strCheckValue) - (i - 1));

          U := SearchForElement(T, strKey);
          Check(U <> Nil, Format('%d.6) Cannot find KEY to check (%s): %s', [Succ(iCheck), GetElements(U, T), strCheckValues[iCheck]]));
          CheckEquals(strCheckValue, U.AsString(True, False), Format('%d.7) Value check failed: ', [Succ(iCheck), strCheckValues[iCheck]]));
          CheckEquals(strValueScope, GetEnumName(TypeInfo(TScope), Ord(U.Scope)), Format('%d.8) Incorrect Scope: %s', [Succ(iCheck), strCheckValues[iCheck]]));
        End Else
          Check(strCheckValue <> '', Format('%d.1) strCheckValue is NULL!', [Succ(iCheck)]));
  Finally
    P.Free;
  End;
End;

Below is an example of the above method being implemented in a test function.

Procedure TestTPascalModule.TestGenericTypeConstructor;

Begin
  TestGrammarForErrors(
    TPascalModule,
    strUnit,
    'type'#13#10 +
    '  TSomeClass<T: constructor> = class'#13#10 +
    '    function GetType: T;'#13#10 +
    '  end;',
    'function TSomeClass<T>.GetType: T;'#13#10 +
    'begin'#13#10 +
    '  Result := T.Create;'#13#10 +
    'end;'#13#10 +
    'procedure Test;'#13#10 +
    'begin'#13#10 +
    '  SomeClass := TSomeClass<SomeObject>;'#13#10 +
    'end;'#13#10,
    [ttErrors, ttWarnings],
    [
      'Types\TSomeClass<T>|TSomeClass<T> = Class|scPublic',
      'Types\TSomeClass<T>\Methods\GetType|Function GetType : T|scPublished',
      'Implemented Methods\TSomeClass<T>\GetType|Function GetType : T|scPublished',
      'Implemented Methods\Test|Procedure Test|scPrivate'
    ]
  );
End;

Well the next phase of testing is to finish fixing the 3 routines which fail and then using another testing harness (see the below image) to throw the parser at as much code as possible looking for Errors and Warning. Obviously the biggest test will be the Berlin source code. The test harness will tell me all the Errors, Warnings and Hints for each module parsed and I'll look to create new test cases for each and fix them. The ultimate goal is for the parser to be able to parser all the Berlin code without any errors.

Additionally I need to change the options dialogue to place these in the IDEs options dialogue a la Chapter 17: Options Page(s) inside the IDE’s Options Dlg, refactor the wizard interfaces a la RegisterPackageWizard and try and get the syntax highlighters to register their own editor options page (whcih used to AV the IDE).

I hope this article provides some food for thought for people new to Unit Testing. I'm sure some will disagree with this approach which is okay, it just allows me to test a lot more information in a quicker time. Since the tests only take seconds to run, fixing one issue and pressing F9 quickly leads to either a passed test or the next failure in that code element.

Next I'm going to have a bit of a break for a few weeks and record some music which I've been trying to do for a while (I seem to write guitar parts I can't play 🙁 ).