Tuesday, October 14, 2014

AAWS - Targeted Error Handling

While troubleshooting AAWS issues (Admission Applications Web Services delivered in Campus Solutions) in a Production Environment, I found the following options that might come in handy.

Let us take AAWS - Submit Application (SAD_SUBMITAPPL) service operation as an example for this blog. Note: The same approach could be extended to any other AAWS service operation.

The problem or difficulty with troubleshooting this service operation is that the faults generated are very generic in nature and does not help with identifying the root cause of the problem.

Here is an example of a fault message generated by SAD_SUBMITAPPPL:

<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Body>
    <SOAP-ENV:Fault>
      <faultcode>Client</faultcode>
      <faultstring>An Error occurred processing this request (14098,286)</faultstring>
      <detail>
        <MSGS>
          <MSG>
            <ID>14098-206</ID>
            <DESCR>Error during Application submit, Contact the Administrator</DESCR>
            <MESSAGE_SEVERITY>I</MESSAGE_SEVERITY>
            <PROPS/>
          </MSG>
        </MSGS>
      </detail>
    </SOAP-ENV:Fault>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

While I can understand why AAWS faults are generic, potentially to mask the nasty error messages from being displayed to the end user, it is my personal opinion that this type of fault message is unacceptable especially for a high volume web service such as AAWS (from a service maintenance point of view). Let us explore how we can put controls in place to be able to identify the root cause for this error in real-time.

Using Delivered Campus Service Oriented Architecture Logging (Doc ID 1511543.1):

This document details couple of options.
  1. Setting up the regular logging at the routing level. This does not really add much value in the case of AAWS.

    Logging:



    Results:



    You will notice that there are no errors logged for the synchronous transaction.

  2. Setting Log Type and Threshold using delivered Campus Service Oriented Architecture (SOA) Logging.



    We can set the Logging Type to Database or File. For this example I chose the Database option. For the Log Threshold we have many options. The 'All' option would provide a lot of detail and something like 'Severe' option would probably give us what we want (in terms of errors).

    The result of this logging should be available in the PS_SCC_INTEG_LOG table. Use the following SQL to get the most recent entries first:

    select * from PS_SCC_INTEG_LOG order by 1 desc;

    You will notice that SCC_INT_LOG_MSG (where SCC_INT_LOG_LEVEL=2) would contain the error message that we are after. In my case, I found that my error message was as follows:

    "Transaction Manager Exception: Error Saving: XXX_CUSTOM_STG tempid: 579 (0,0) SCC_COMMON.ENTITY.StagedEntity.OnExecute  Name:save  PCPC:16872  Statement:322 Called from:SCC_COMMON.ENTITY.ChildEntity.OnExecute  Name:save  Statement:101 Called from:SCC_C"

    Now, this would tell us exactly why and where (peoplecode statement) the error occurred. This helps greatly in comparison to the Generic Error Message (Error during Application submit, Contact the Administrator).

    While this is very useful for short term/unexpected troubleshooting, one downside of the Campus SOA Logging is that it is not specific to a service operation. So if we turn on logging it would take effect for all delivered service operations that are represented using the Campus SOA. You will notice that if we have this logging turned on in a Production Environment then the table PS_SCC_INTEG_LOG would quickly fill up. Another scenario where this logging might fall short is if we want to send back real-time meaningful response/fault messages for the consumer to process and take action.
Now that we have exhausted delivered options (please correct me if I missed anything!), let us look at a targeted error handling approach (returning appropriate exceptions in the fault message) involving a minor customization. Note: While customizations would involve cost and maintenance considerations, it is generally the case of whether the benefits outweigh the effort and maintenance. I will let you make the decision based on the circumstances in your organization.

AAWS Targeted Error Handling - Customizing the Handler:

The OnRequest Handler code for the AAWS - SAD_SUBMITAPPL is implemented in the following class and method:
Class: SCC_OLA.HANDLERS.Admissions.OnExecute
Method: submitApplication

Let us take a look at a snippet of the delivered code that deals with part of the error handling which I customized:


Here is the same snippet of delivered code with my customization:


Let us examine my customization in detail:

         /* XXX - SV - 20141013 - Test Start */

         Local SCC_COMMON:ENTITY:LOG:MessageEntry &msgEntryCust = create SCC_COMMON:ENTITY:LOG:MessageEntry();
         Local array of string &parms = CreateArrayRept("", 0);
         &parms [1] = &e2.ToString();
         &msgEntryCust.DataPopulateV1(14098, 181, &msgEntryCust.Severity_Error, &parms, "", "", "");
         &returnError.writeEntry(&msgEntryCust);

         /* XXX - SV - 20141013 - Test End */

I made use of the MessageEntry class to create another message detail (<MSG> section), populate the message detail with the exception information and append it to the &returnError object which is the fault message.

Note: I used the same message set number and message number (14098, 181) that is referenced in the logError method right at the start of this exception handling section. 
&p_logger.logError(MsgGet(14098, 181, "", &e2.ToString()));

Let us see the results of this customization.

<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> 
  <SOAP-ENV:Body> 
    <SOAP-ENV:Fault> 
      <faultcode>Client</faultcode> 
      <faultstring>An Error occurred processing this request (14098,286)</faultstring> 
      <detail> 
        <MSGS> 
          <MSG> 
            <ID>14098-206</ID> 
            <DESCR>Error during Application submit, Contact the Administrator</DESCR> 
            <MESSAGE_SEVERITY>I</MESSAGE_SEVERITY> 
            <PROPS/> 
          </MSG> 
          <MSG> 
            <ID>14098-181</ID> 
            <DESCR>Transaction Manager Exception: Error Saving: XXX_CUSTOM_STG tempid: 583 (0,0) SCC_COMMON.ENTITY.StagedEntity.OnExecute  Name:save  PCPC:16872  Statement:322
Called from:SCC_COMMON.ENTITY.ChildEntity.OnExecute  Name:save  Statement:101
Called from:SCC_COMMON.ENTITY.StagedEntity.OnExecute  Name:saveChildren  Statement:374
Called from:SCC_COMMON.ENTITY.StagedEntity.OnExecute  Name:save  Statement:303
Called from:SAD_ADM_APPL.ApplicationManager.OnExecute  Name:saveApplication  Statement:70
Called from:SCC_OLA.HANDLERS.Admissions.OnExecute  Name:submitApplication  Statement:595
Call
</DESCR> 
            <MESSAGE_SEVERITY>E</MESSAGE_SEVERITY> 
            <PROPS/> 
          </MSG> 
        </MSGS> 
      </detail> 
    </SOAP-ENV:Fault> 
  </SOAP-ENV:Body> 
</SOAP-ENV:Envelope>

Highlighted in orange is the marker for the custom message that was added and highlighted in green are the details of the custom message.

You can see how we can improve the details in the fault messages enabling both technical and functional staff to quickly identify the problem with the application and take corrective action. This is very effective especially when there could be application submission peaks where we can expect thousands of application requests in a day.

Note:
  • Please test (test and test!) this thoroughly before applying the suggested customization to your Production Environment (if you think this might be useful).
  • In the submitApplication method used in this example there are a 4 occurrences of &returnError.writeEntry method being invoked. It is recommended that each of those are appropriately extended using the approach in the sample code and example provided in this blog.
  • Customization Environment Details: Campus Solutions 9.0 (Bundle 34) and PeopleTools 8.52.22.

14 comments:

  1. Replies
    1. @Pallavi Thanks for the feedback! Appreciate it.

      Delete
  2. Hi,
    Is it required to write for all the four. or only this is enough?

    Thanks

    ReplyDelete
  3. @Anonymous - If you want to use the above code to help with troubleshooting of the submitApplication then I would suggest that you address all the four references to &returnError.writeEntry in the method.

    You have to use the sample code provided and adjust accordingly.

    Note: This example only talks about submitApplication. You could also extend the same logic to other operations like createApplication, saveApplication, getApplication, etc.

    Thanks!

    ReplyDelete
  4. One method we used it to log errors is using log file in integration logging and use Logger class to log service operation level login if required. Obviously this has been used while development (and can be deployed to production as well, downside is logging overhead in prod environments)

    to do that we have to import UTIL class in SCC_SERVICE

    import SCC_SERVICE:UTIL:*;

    and use something like

    rem Debug Statement;
    Local SCC_SERVICE:UTIL:Logger &p_logger = create SCC_SERVICE:UTIL:Logger("Debug Service oper", "Submit appls");
    &p_logger.logFine("this is the error message " | e2.Tostring()) ;

    ReplyDelete
    Replies
    1. @Ayesha Wee - Thanks for your input.

      Yes. Definitely, that is an option for more detailed logging for troubleshooting scenarios. Also, the other good thing about Logger class is that we can suppress certain levels of logging by updating the log level setting.

      Delete
  5. This help me out a lot. I added the mod to the 4 catch routines and it returned the error message with much greater detail. I was able to fix the issue quickly. This is awesome and it got me out of a jam. Thank you so much! BTW, this mod works just fine in tools 8.53.

    ReplyDelete
    Replies
    1. @Jack - Thanks! Good to hear that it works fine in 8.53 as well.

      Delete
  6. This is very helpful, We are facing same type of error as mentioned below, can you please help to identify why we are facing this error message on few application, I have given my email addrr:




    Client
    An Error occurred processing this request (14098,286)



    14098-199
    Error during Application save, Contact the Administrator
    I






    ReplyDelete
    Replies
    1. Hi Sameer - I could not tell you why you are receiving this error.

      I would suggest that you take the request message that caused this error and use either the Handler Tester or Service Operation Tester to run a PeopleCode/SQL trace. This is the best way to figure out why you are receiving such errors.

      PeopleTools > Integration Broker > Service Utilities > Handler Tester, Service Operation Tester

      Delete
  7. Dear All:
    I am running peopleSoft CMS. Now a days I am facing following error
    Open of file C:/TEMP failed: A file or directory in the path name does not exist.. (2,633) SCC_SERVICE.UTIL.FileLogStrategy.OnExecute Name:open PCPC:679 Statement:12
    Called from:SCC_SERVICE.UTIL.Logger.OnExecute Name:Open Statement:36
    Called from:SCC_SERVICE.AbstractServiceOperation.OnExecute Name:setupLog Statement:89
    Called from:SCC_SERVICE.AbstractServiceOperation.OnExecute Name:execute Statement:11
    Called from:ADM_APPL_ENTRY1.GBL.SavePostChange Statement:13 The open could not be completed because of the error.

    any solution ?

    ReplyDelete
    Replies
    1. It appears that there is an error while writing/creating the log file. Check if your application is able to write to C:/TEMP. You might need to work with your ps/sys admins.

      Delete
  8. hi sasank,
    Our system is setup such that a third party sends student application data in real time to a custom table in peoplesoft on our system. Could i wire that data coming into the custom table to the AAWS web service to create a student application.

    ReplyDelete
    Replies
    1. Hi Alex - This is a strategy that will work.

      Once the data is in your custom table, you could write an app engine (batch process) to pick up the data and process it using the AAWS service. I don't have examples but I have seen this working in practice.

      Delete