How to implement paging for ECMA2 when data source doesn’t support paging

During my last FIM project in Cairo, one of my colleagues asked me how to deal with hardcoded object limit in ECMA2. It appeared later on that problem is not the object limit but the page size versus data source which doesn’t support paging. I promised to find code example for that case and didn’t done that by now. So here it is (I hope still useful for someone).

How to implement paging in ECMA2 when data source doesn’t support paging? Read the data from the data source as it is possible. When you will see that you have enough records for the data page stop reading (and set flag that you didn’t finish). FIM will call your data reading code again (and again if needed). The only thing you have to remember is to keep some global variables in case you have to pass connection information or counters.

Here are examples of implementation for various situations (depending on methods available to read data).

Table like data interface

Let’s assume you have data source were there are following methods to access identity data:
Int GetPeopleCount() – return number of identities
Person[] GetAllPeople() – return enumerable structure with identity objects

Code example:

//global variables

private int m_importPageSize;
private Person[] allUsersList;
int userToRead;

public OpenImportConnectionResults OpenImportConnection(KeyedCollection<string,
  ConfigParameter> configParameters,
  Schema types,
  OpenImportConnectionRunStep importRunStep)
{
  m_importPageSize = importRunStep.PageSize;
  userToRead = 0;

  //Get list of all users (or other identity objects)
  allUsersList = GetAllPeople();

  return new OpenImportConnectionResults();
}

public GetImportEntriesResults GetImportEntries(GetImportEntriesRunStep importRunStep)
{
  GetImportEntriesResults importReturnInfo;
  List<CSEntryChange> csentries = new List<CSEntryChange>();

  // Full Import.

  if (OperationType.Full == m_importOperation)
  {
    if (allUsersList != null)
      while (userToRead < GetPeopleCount() &&
      csentries.Count < m_importPageSize)
      {
        try
        {
          CSEntryChange cs = CSEntryChange.Create();
          cs.ObjectModificationType = ObjectModificationType.Add;
          cs.ObjectType = "Person";

          //read data from object representing Person
          // for example:
          cs.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("ObjectName", AddCR(allUsersList[userToRead].Object_Name)));

          csentries.Add(cs);
        }

        catch (Exception e)
        {
          throw new ServerDownException("Read error (user: " + allUsersList[userToRead].Object_Name.ToString() + ")", e);
        }

        userToRead++;
      }
    }

  // Create the result
  importReturnInfo = new GetImportEntriesResults();
  if (allUsersList != null)
  {
    importReturnInfo.MoreToImport = (userToRead < GetPeopleCount());
  }
  else
    importReturnInfo.MoreToImport = false;

  importReturnInfo.CSEntries = csentries;

  return importReturnInfo;
}

Stream like data interface

Following methods available to access identity data:
DataConn OpenConnection(…) – method returning handle to data connection/stream
Person GetNextPerson(DataConn d) – method returning next identity object from stream
Bool EndOfData(DataConn d) – method to check if there is something more to read

Code example:

 
//global variables
private int m_importPageSize;
private DataConn dataConnection;

public OpenImportConnectionResults OpenImportConnection(
                                                KeyedCollection<string, ConfigParameter> configParameters,
                                                Schema types,
                                                OpenImportConnectionRunStep importRunStep)
{
  m_importPageSize = importRunStep.PageSize;

  //Open data connection
  dataConnection = OpenConnection(…);

  return new OpenImportConnectionResults();
}

public GetImportEntriesResults GetImportEntries(GetImportEntriesRunStep importRunStep)
{
  GetImportEntriesResults importReturnInfo;
  List csentries = new List();

  // Full Import.
  if (OperationType.Full == m_importOperation)
  {
    while (!EndOfData(dataConnection) &&
           csentries.Count < m_importPageSize)
    {
      try
      {
        Person person = GetNextPerson(dataConnection);

        CSEntryChange cs = CSEntryChange.Create();
        cs.ObjectModificationType = ObjectModificationType.Add;
        cs.ObjectType = "Person";                               

        //read data from object representing Person
        // for example:
        cs.AttributeChanges.Add(AttributeChange.CreateAttributeAdd("ObjectName", AddCR(person.Object_Name)));

        csentries.Add(cs);
      }
      catch (Exception e)
      {
        throw new ServerDownException("Read error (user: " + person.Object_Name.ToString() + ")", e);
      }
    }
  }

  // Create the result
  importReturnInfo = new GetImportEntriesResults();

  importReturnInfo.MoreToImport = (!EndOfData(dataConnection));

  importReturnInfo.CSEntries = csentries;

  return importReturnInfo;
}