By Chris Malek | 01 June 2011
Understanding Integration Broker Queue Partitioning
This article will explain Message Queue partitioning in PeopleTools Integration Broker.
Partitioning is defined at the Queue level and it tells the integration broker if messages in a queue can be processed in parallel or not. This is queue dependent and setting up proper queues and partitioning can be critical in setting up effective loosely coupled systems. If you configure it incorrectly, one error in the queue can block everything which can be disastrous for high volume message queues. There are times that you want to configure partitioning and other times when you do not. This is really dependent on the nature of the message and how the receiving system handles the data.
Partitioning is best explained with an example.
USER_PROFILE Queue Partitioning Example
Lets take the case of the USER_PROFILE_SYNC service operation. This Service Operation is used to sync PeopleTools user profiles between different PeopleTools databases or other systems. This service operation is defined in the USER_PROFILE queue.
When you look at the queue setup for the USER_PROFILE queue you can see that all the common fields are listed on the right hand side. In this screenshot the OPRID is checked for partitioning.
- Lets say that 3 users all change their password in your HR database and the USER_PROFILE_SYNC service operation fires to synchronize that user change with your Finance Database.
- Let’s assume that user JOHND, SALLIES, and ROBERTD all changed their password in that order but within seconds of each other. The user JOHND hit save first so the USER_PROFILE_SYNC message for his user profile was queued up for publication to your finance database.
- Let’s say that that the JOHND user id did not have permission to publish user profile messages to finance due to a security setup on his profile.
Since we have partitioning setup on the OPRID common field level, the publication error on the JOHND user profile will NOT block the publication messages of SALLIES and ROBERTD. The SALLIES and ROBERTD message both went to success. This is shown in the drawing where each user basically has their own “lane” in the queue and the “lane” is setup based on the partitioned field.
If JOHND decides that he does not like his new password and changes it a minute later then that second password change will actually sit in “NEW” status behind the message in error because the queue is partitioned and the integration broker was smart enough to realize that there was a message for that user in error and it does not processes the second or third message. All messages for the JOHND user are basically frozen but other user profiles are not impacted. If someone were to come in and cancell the JOHND message that was in error status the second JOHND message would actually try to publish and would error out if the security situation had not been corrected.
What would happen if we did not have any partitioning on the USER_PROFILE queue? If nothing was checked on the right hand side of the queue setup (i.e. No Partitioning), then the publication error on the JOHND message would actually completely freeze all other user profiles publications to finance which is probably not a great idea for user profiles. This is shown in the drawing where there is basically one “lane” in the queue and one error holds up all other messages.
So in the case of the USER_PROFILE queue, it is a good idea to actually setup partitioning because user profiles tend to be independent of one another.
You really have to understand the business process that creates the messages in the publishing system and also know how the subscribing system handles them. There is one binary answer regarding when to turn partitioning on.
When to setup Queue Partitioning
Generally when you have messages that can process independently of each other you should setup partitioning. Setup tables are generally good candidates for this.
When NOT to setup Queue Partitioning
If you need the messages to process in the order they were published then you do NOT want to turn partitioning on.
Additionally, if all the service operations in a queue do not have any common fields then you can partition by Service Operation Name, Publishing user or the publishing process name.