Contents
Introduction
Concurrent Managers
Internal Concurrent Manager
Product Concurrent Managers
Control of the Concurrent Managers
Tuning Parameters Install Issues
Passwords
Demo Database
Fresh Install or Upgrade
USDSOP failed during spawn of INVLIBR process
Migration
Startup of Concurrent Manager Failed
Are the Concurrent Managers Down?
The Concurrent Managers are Down
10.7 NT Concurrent Managers
Errors when Starting the Managers on NT
Concurrent Manager Tables
FND_CONCURRENT_PROCESSES
FND_CONCURRENT_REQUESTS
FND_CONCURRENT_QUEUES
Concurrent Managers
Internal Concurrent Manager
Concurrent Managers are composed of two different types of managers. One type is the Internal Concurrent Manager (ICM). The ICM controls all of the other concurrent managers. One of its' most important jobs in life is to ensure that the other concurrent managers are up and running. This is accomplished when the ICM wakes up and that is normally every 20 minutes. This will be discussed later in the paper. The ICM also performs the function of applying any incompatibility rules. Starting in 10.7, this can be done by the Conflict resolution Manager. This is where one program can not run with another. The other types of concurrent manager, are product concurrent managers (this is my phrasing).
The ICM logfiles will be located in either $FND_TOP/$APPLLOG or $APPLCSF/$APPLLOG ($APPLLOG is normally set to log). The file name is normally "std.mgr" but the name could also be "SID.mgr". When starting up the ICM, a parameter called "mgrname" can be used to rename the logfile.
APPLCSF is an optional parameter that can come in handy. The reasoning for APPLCSF, you have multiple APPL_TOPs and want to keep all of your logfiles and output files in the same place. Also, if APPLCSF is set all of the files (logfiles and output) for your products will be placed in this directory. This is convenient but never necessary. APPLCSF is set within the environment file on APPL_TOP.
Product Concurrent Managers
During the course of this paper specialized concurrent managers will consist of all concurrent managers with the exception of the ICM (e.g. Standard, Inventory, MRP, PA, and any user defined managers). These are the Concurrent Managers that run your jobs. They run concurrent requests, reports and processes. If the variable of $APPLCSF is NOT set the log and output files will be in each Product Top directory. All of the manager logs will be in either $FND_TOP/$APPLLOG or $APPLCSF/$APPLLOG ($APPLLOG is normally set to log).
Control of the Concurrent Managers
The ICM can be controlled from the operating system or within the applications.
This is the only manager that can be controlled from the operating system. When all of the concurrent managers have to be shutdown, shutting down the ICM is the way to do it. When the ICM is shutdown it will automatically shut down any running concurrent managers. Likewise, when the ICM is started it will restart any concurrent manager that were enabled (before shutdown) and the "target processes > 0" when the ICM was shutdown. Again this can be accomplished from either the o/s or within applications. If a product manager or the standard manager was deactivated prior to the ICM coming down, it will have to be restarted manually after the ICM comes back up.
In Char Mode (under SYSADMIN) go to: \Nav Con Man Admin
In GUI (under SYSADMIN) go to: Concurrent Manager Administrator
From the command line the following will bring down the ICM:
10.6 and prior Releases
CONCSUB applsys/<password> SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ABORT
For 10.7 CONCSUB apps/<password> SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ABORT
The ICM can also be started for within the applications and from the command line. From the applications do the following:
In Char Mode (under SYSADMIN) go to: \Nav Con Man Admin
In GUI (under SYSADMIN) go to: Concurrent Manager Administrator From
the command line the following will start the ICM. If you are
experiencing problems, you can add "diag=y" to the end of the command
line.
10.6 and prior Releases
startmgr sysmgr=applsys/<password> mgrname='SID'
For 10.7 startmgr sysmgr=applsys/<password> mgrname='SID'
The parameters startmgr & sysmgr are mandatory.
mgrname is if you want to name your concurrent managers.
If this option is chosen then the ICM logfile will now be the %lt;name%gt;mgrname.mgr.
diag - will create a more verbose logfile.
This option is normally used to troubleshoot ICM problems.
Other parameters like sleep, pmon, quesiz are described on pg. 8-18 of the System Administration Reference Manual Release 10. Tuning Parameters
Internal Concurrent Manager
There are three parameters that affect the performance of the ICM. These parameters are sleep time, pmon cycle, and queue size.
First, Sleep time. This default is 60 seconds. This is the number of seconds that the internal concurrent manager waits between times it looks for new concurrent requests.
Second, PMON. This is the number of sleep cycles the ICM waits between time it checks for failed concurrent managers. The default is 20.
Finally, Queue size. This is the number of PMON cycles that the ICM waits between times it checks for new or disabled concurrent managers. The queue size default is 1.
The above parameters can be set during startup of the ICM. The parameters pmon and quesiz can be accomplish if the user does a 'Verify' on the ICM within applications. The path is \Nav Con Man Admin
Product Managers
The sleep time for the product managers are set to the default of 60 seconds. This is the number of seconds that the manager waits between checking the list of pending concurrent requests. The sleep parameter is modified in the 'Work Shifts' zone of the 'Define Concurrent Manager' screen. \Nav Con Man Def
The last tuning parameter is called 'Cache Size'. The Cache Size is the number of concurrent requests that the manager picks up from the FND_CONCURRENT_REQUST table when the manager wakes up. For customer defined managers the Cache Size is either blank or set to 0. This number should be set to twice the number of target processes. If the target process is set to three (\N C M A) then the Cache Size should be six.
The reasoning is, if the manager has three target processes it could run three jobs. But if it only picks up one job every 60 seconds then you are losing valuable processing time. If it picks up six jobs then the managers will continuously be processing. This parameter can ONLY be set though the GUI. It can not be set from within character mode.
Install Issues
Passwords
10.7 the APPLSYS and APPS passwords MUST be the same. The user can change the default passwords, but both passwords MUST be the same. The problem is functionality will be lost if the passwords are different. You will be able to start the concurrent managers, but other items (i.e. adadmin, adpatch) will not work correctly. If the user decides to change the passwords they MUST follow the steps in GSX 1009719.6 - Resetting Applications Passwords (GSX is Oracles Global Solution Exchange)
Demo Database
When installing the 10.7 demo database the user must be aware that they might need to update a couple of tables for the concurrent managers to come up properly.
The reasoning is these are compressed .dbf files and the data that is contained within them is from another system. The following tables need to be updated:
(This is how we cleanup the FND tables)
truncate table FND_CONCURRENT_REQUESTS
truncate table FND_CONCURRENT_PROCESSES
After the above have been updated start the ICM in the diagnostics state. This will give you a more verbose logfile.
10.7
startmgr sysmgr=apps/<password> diag=y
10.6 and prior
startmgr sysmgr=applsys/<password> diag=y
When the managers come up look at the logfile 'std.mgr'. On the first page you should see (if you used diagnostics):
Process Monitor (PMON) Method is LOCK
If you do NOT see LOCK do the following. It is always a good idea to check with WorldWide Support before changing the PMON method. The above will set the PMON method to LOCK. Now restart the ICM.
The PMON method gives the customer three choices and they are:
OS = The internal concurrent manager determines whether or not a process is still running by sending a kill -0 to the process. This is merely a test to see if the process is still here. (THIS METHOD IS NO LONGER USED.)
RDBMS = The internal concurrent manager queries against v$process to retrieve the session id (the one that started the request) and the operating system process id. The reason it needs to query against both columns and not just the OS process id (pid) is because if there are concurrent managers running on two different clients against the same RDBMS then two requests could have the same OS pid.
LOCK = The internal concurrent manager tries to get a lock on the process it is monitoring. The name of the lock is determined by a sequence (which is the id of the individual managers (standard, etc)) and the program in question. Once the manager is able to get the lock, then it knows the process is no longer running. Two installs in one RDBMS share the lock method because if the two managers get the same sequence id (one in each instance) then the second manager will never be able to start because the ICM will always think its already running.
(THIS IS THE RECOMMENDED METHOD FROM 10.6 ON.) Caution: If using DCP, you can only have one system using LOCK. Others must be set to rdbms.
If you are still getting errors, check if you have any invalid objects in your database that start with FND_CON% owned by APPS or APPLSYS. If you have invalid objects recompile them, they could be causing the problem. If the packages will NOT recompile the packages will need to be recreated. Please contact support to help find and recreate the packages.
Fresh Install or Upgrade
With a fresh install you will not have to cleanup the concurrent manager tables (they have just been installed). Once you have tried to start the ICM go to the logfiles and check for errors. It is possible to find the following:
If you find the following -- Please note the following are NOT all of the possible errors you could receive - For a more comprehensive list of errors please see GSX 1011248.6 What are the common concurrent manager issues?:
If the manager fails to start please go to that section:
USDSOP failed during spawn of <Product>LIBR process
The above normally happens when an application is installed as shared. The seed data has been placed into FND_CONCURRENT_QUEUES and there is no executable on the file system. This is because the install has enabled that manager and the ICM cannot find the corresponding executable. The best example is if '<Product>LIBR' is getting the USDSOP error and <Product> was not in installed or it is installed as shared.
The fix for this problem is to set the target processes to 0 in 'Define Concurrent Managers' screen.
In Char Mode (under SYSADMIN) go to: \Nav Con Man Define In GUI (under SYSADMIN) go to: Concurrent Manager Define APP-01167 - Internal Concurrent Manager has encountered an error:
This is a generic error which may be followed by an Oracle error or:
APP-01054 - Distributed concurrent manager enabled, lock PMON method requested
APP-01114 - AFPCAL received failure code while running FNDCPMGR
Solution: Make certain that $APPLDCP is set to off or, if using distributed concurrent processing, ensure the PMON method is LOCK. (Only on primary node. Other nodes should use rdbms.)
Migration
During a migration you may find the same type of problems as in the DEMO install. The reasoning is, you are moving from one platform to another. If you have not moved your file system the ICM may not come up. The fix may be as simple as cleaning up the concurrent manager tables. It is possible that the concurrent manager is looking for files that were on the other system.
Start Up of Concurrent Manager Failed
Now we are getting to point where users might be breathing down your neck. You need to ask a few basic questions: Are the Concurrent Managers Down? Sounds
like a stupid question. BUT, from within your applications do the
target process equal the number of actual processes? If they do then
technically your managers are not down. Now from the operating system do
you see FNDLIBR processes when you do a:
ps -ef | grep FNDLIBR
Are the number of FNDLIBR processes equal to 1 + the number of target processes from within your applications?
Please Note: If you have more than set of applications running on the system then you WILL get more back from your grep. But, if these are your only applications then you should not get anything but the grep back.
If the managers are not down submit a concurrent request to bring them down:
CONCSUB applsys/<password> SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ABORT
If this request only goes to pending never runs then we will need to kill the concurrent managers from the command line.
The Concurrent Managers are Down
1) Let's go to the logfiles. Remember they are in either $APPLCSF/$APPLLOG or $FND_TOP/$APPLLOG.
2) Rename the logfile. The reasoning here is, the ICM only appends to the log and it is much easier to start with a fresh log.
Normally
this file is called "std.mgr" but if you name your concurrent managers
by using the "mgrnam" parameter while starting the ICM then the logfile
will normally be "SID.mgr". select count(*) from fnd_concurrent_requests where status_code='T'; select count(*) from fnd_concurrent_requests where status_code='R' You can remove all of the completed requests - This will reduce the size of the FND_CONCURRENT_REQUESTS table. The
Next update statements should never be done without the help of support
or development. It is possible to cause data corruption. truncate table FND_CONCURRENT_PROCESSES; 10.7
startmgr sysmgr=apps/<password> diag=y
10.6 or prior
startmgr sysmgr=applsys/<password> diag=y In GUI: Concurrent Manager Administer
In Char: \Nav Con Man Admin Select count(*) from dba_objects where status='INVALID'; Process Monitor (PMON) Method is LOCK
If you do NOT see LOCK do the following. It is always a good isea to check with WorldWide Support before changing the PMON method. The PMON method gives the customer three choices and they are:
OS = The internal concurrent manager determines whether or not a process is still running by sending a kill -0 to the process. This is merely a test to see if the process is still there. (THIS METHOD IS NO LONGER USED.)
RDBMS = The internal concurrent manager queries against v$process to retrieve the session id (the one that started the request) and the operating system process id. The reason it needs to query against both columns and not just the OS pid is because if there are concurrent managers running on two different clients against the same RDBMS then two requests could have the same OS pid.
LOCK = The internal concurrent manager tries to get a lock on the process it is monitoring. The name of the lock is determined by a sequence (which is the id of the individual managers (standard, etc)) and the program in question. Once the manager is able to get the lock, then it knows the process is no longer running.
Two installs in one RDBMS share the lock method
because if the two managers get the same sequence id (one in each
instance) then the second manager will never be able to start
because the ICM will always think its already running.
(THIS IS THE RECOMMENDED METHOD FROM 10.6 ON.)
10.7 NT Concurrent Managers:
There are a few interesting issues with concurrent managers on NT. The concurrent managers are services on the NT. They have to be added and removed with the following commands.
TO Add: (pg. 5-9 NT Install Manual) C:\> cmsrvadm add <APPL_CONFIG> [automatic | manual]
Note: When adding the service you MUST type in the UserID and password. It is does NOT default. If you do NOT type in the password the you will get an ORA-1017 error in the event viewer log.
To Remove: (pg. A-5 NT Install Manual) C:\> cmsrvadm remove <APPL_CONFIG>
Normally the <APPL_CONFIG> will be either APPLSYS or APPDEMO. One point, only the user who added the service will be allowed to remove the service. After the service has been added then it needs to be started from 'Services' in the 'Control Panel'.
Errors when Starting the Managers on NT
Now if your like most people, at some point you'll get an error when starting the Concurrent Managers. The normal generic error from the 'Services' goes something like this:
At the command line:
Could not start the OracleConcMgr<APPL_COMFIG> service on 'Machine Name'
Error 2140: An internal Windows NT error occurred.
(The above error doesn't say a whole lot - to get a better error message go to: Start - Programs - Administrative Tools - Event Viewer Go to Log and Click on Application)
Find the Source of OracleConcMgr<APPL_CONFIG>
The Event Log Says:
The Description for Event ID (0) in Source [OracleConcMgr<APPL_CONFIG>] could not be found. It contains the following insertion string(s): Cannot connect to database. Sid=ORACLE_SID ORA error 12154.
The fix:
Add a loop-back in your tnsnames with the ORACLE_SID as the alias.
ANOTHER ERROR:
At the command line:
Service not started. Check the event log for more information.
Exit Code: 0
Service Specific Exit Code: 0
The Event Log Says:
The Description for Event ID (0) in Source [OracleConcMgr<APPL_CONFIG>] could not be found. It contains the following insertion string(s): OracleConcMgr<APPL_CONFIG> error 3: The system cannot find the path specified.
Cannot delete Concurrent Manager logfile.
\%FND_TOP%\logdemo\CM_LOG. Please check that these are no processes running.
The Fix:
Check to ensure the directory 'LOGDEMO' is in FND_TOP. On the NT even if APPLCSF is set, the concurrent manager log files will still write to FND_TOP/log.
ANOTHER ERROR:
At the command line:
Service not started. Check the event log for more information.
Exit Code: 0
Service Specific Exit Code: 0
The Event Log Says:
The Description for Event ID (0) in Source [OracleConcMgr<APPL_CONFIG>] could not be found. It contains the following insertion string(s): Cannot connect to database SID = <ORACLE_SID> ORA error 1017.
The Fix:
One of the following were typed incorrectly:
APPS schema's UserID:
APPS schema's password:
LAST ERROR I'LL LIST:
At the command line:
Failed to start Service OracleConcMgr<APPL_CONFIG>
Error: 1069 The service did not start due to a logon failure
The Event Log Says:
No event log for this error
The Fix:
The 'Windows NT user account password' was typed incorrectly:
Remove the service and add it back.
Concurrent Manager Tables
The following tables are the most important tables for the concurrent managers. These are not the only tables that have to do with the concurrent managers.
FND_CONCURRENT_PROCESSES
Stores information about concurrent managers. Each row includes values that identify the ORACLE process, the operating system process, and the concurrent manager (QUEUE_APPLICATION_ID and CONCURRENT_QUEUE_ID). You need one row for each instance of a running concurrent manager (each process), as well as one row for the Internal Concurrent Manager. AOL uses this table to keep a history of concurrent managers. You should never update this table manually.
Purge Concurrent Requests and /or Managers Data program to delete history information periodically.
FND_CONCURRENT_REQUESTS
Stores information about individual concurrent request. Each row includes values that identify the particular request and its parameters, such as who submitted it, the request type, whether the request should run sequentially with other requests in the same logical database (SINGLE_THREAD_FLAG), whether the request is on hold (HOLD_FLAG), whether to display the request in the View Requests form for the request submitter to review, and what status a phase the concurrent request is in. Each row also includes values that identify the concurrent program, its execution and argument methods, and whether the program is constrained (QUEUE_METHOD_CODE). Each row also includes flags that indicate the requests, priority related to other requests, as well as values that specify how the concurrent manager should print program output, if any.
ARGUMENT1 through ARGUMENT25 contain any arguments the application passes to the concurrent program. If the concurrent program needs more that 25 arguments to run, the first 25 arguments are stored in this table, ARGUMENT26 through ARGUMENT100 are stored in FND_CONC_REQUEST_ARGUMENTS. ARGUMENT_TEXT contains the concatenation of concurrent requests arguments and COMPLETION_TEXT contains a message about how the request completed. The row also contains dates that the request was submitted, requested to start and actually run. REQ_INFORMATINO is used with reports sets to remember the status of the request between runs. When the request is set to use automatic resubmission, RESUBMITTED is a flag to indicate whether the request has been resubmitted or not.
RESUBMIT_INTERVAL_TYPE_CODE specifies whether to start interval count down from the requested start time or the completion of the request. RESUBMIT_INTERVAL_TYPE_CODE indicates whether interval unit is in Days, Hours, Minutes, or Months. RESUBMIT_TIME set the time of the day to rerun the concurrent request. RESUBMIT_INTERVAL indicates the number of units of time when the identical request will be resubmitted. RESUBMIT_END_DATE is the date the request stops resubmitting itself. IS_SUB_REQUEST is a flag that identifies a child request and HAS_SUB_REQUEST is a flag that identifies a parent request. Each child request also needs to have values in PARENT_REQUEST_ID to show what parent request submitted the child request and PRIORITY_REQUEST_ID to tell what priority the parent request has and what priority the child request should have. You need one row for each concurrent request. Though you should occasionally delete from this table, you should not modify any of its data.
FND_CONCURRENT_QUEUES
Stores information about concurrent managers. Each row includes the name and description of a concurrent manager, as well as values that identify the program library attached to the manager. CACHE_SIZE contains the buffer size (how many requests a concurrent manager should "remember" each time it checks the list of waiting requests), and MAX_PROCESSES determines the maximum number of concurrent requests a manager can run at a time (depends on work shifts). Then manager process automatically updates RUNNING_PROCESSES during start up and shut down time. Each row also includes the time in seconds for the concurrent manager to wait before checking for pending concurrent requests, and a code to activate, deactivate, or reset the manager. You need one row for each concurrent manager defined at your site, with a minimum of one row. AOL uses this information to activate concurrent managers.
Introduction
Internal Concurrent Manager
Product Concurrent Managers
Control of the Concurrent Managers
Tuning Parameters
Passwords
Demo Database
Fresh Install or Upgrade
USDSOP failed during spawn of INVLIBR process
Migration
Startup of Concurrent Manager Failed
Are the Concurrent Managers Down?
The Concurrent Managers are Down
10.7 NT Concurrent Managers
Errors when Starting the Managers on NT
Concurrent Manager Tables
FND_CONCURRENT_PROCESSES
FND_CONCURRENT_REQUESTS
FND_CONCURRENT_QUEUES
Introduction
Concurrent Managers are one of the most important and often the most feared process within Oracle's Applications. When a customer can not get a concurrent manager up and all of their production or testing is backing up, this is a very stressful time. This paper will try to remove some of the mystery and ease some of your fears. It will NOT go over every possible problem. It will also only cover the concepts for UNIX and not VMS or NT. There are other white papers that cover most of the known errors. This paper will also cover some of the basic tuning aspects of concurrent managers. It will NOT cover system-tuning parameters. Then finally, this paper will go over the three most important tables for the concurrent managers. We will cover some of the most common and our solutions for them.Concurrent Managers
Internal Concurrent Manager
Concurrent Managers are composed of two different types of managers. One type is the Internal Concurrent Manager (ICM). The ICM controls all of the other concurrent managers. One of its' most important jobs in life is to ensure that the other concurrent managers are up and running. This is accomplished when the ICM wakes up and that is normally every 20 minutes. This will be discussed later in the paper. The ICM also performs the function of applying any incompatibility rules. Starting in 10.7, this can be done by the Conflict resolution Manager. This is where one program can not run with another. The other types of concurrent manager, are product concurrent managers (this is my phrasing).
The ICM logfiles will be located in either $FND_TOP/$APPLLOG or $APPLCSF/$APPLLOG ($APPLLOG is normally set to log). The file name is normally "std.mgr" but the name could also be "SID.mgr". When starting up the ICM, a parameter called "mgrname" can be used to rename the logfile.
APPLCSF is an optional parameter that can come in handy. The reasoning for APPLCSF, you have multiple APPL_TOPs and want to keep all of your logfiles and output files in the same place. Also, if APPLCSF is set all of the files (logfiles and output) for your products will be placed in this directory. This is convenient but never necessary. APPLCSF is set within the environment file on APPL_TOP.
Product Concurrent Managers
During the course of this paper specialized concurrent managers will consist of all concurrent managers with the exception of the ICM (e.g. Standard, Inventory, MRP, PA, and any user defined managers). These are the Concurrent Managers that run your jobs. They run concurrent requests, reports and processes. If the variable of $APPLCSF is NOT set the log and output files will be in each Product Top directory. All of the manager logs will be in either $FND_TOP/$APPLLOG or $APPLCSF/$APPLLOG ($APPLLOG is normally set to log).
Control of the Concurrent Managers
The ICM can be controlled from the operating system or within the applications.
This is the only manager that can be controlled from the operating system. When all of the concurrent managers have to be shutdown, shutting down the ICM is the way to do it. When the ICM is shutdown it will automatically shut down any running concurrent managers. Likewise, when the ICM is started it will restart any concurrent manager that were enabled (before shutdown) and the "target processes > 0" when the ICM was shutdown. Again this can be accomplished from either the o/s or within applications. If a product manager or the standard manager was deactivated prior to the ICM coming down, it will have to be restarted manually after the ICM comes back up.
In Char Mode (under SYSADMIN) go to: \Nav Con Man Admin
In GUI (under SYSADMIN) go to: Concurrent Manager Administrator
- You need to query the concurrent manager record, by doing a "Query Run".
- In the Control column either type 'Terminate' or 'Deactivate' or do a QuickPick and then save the screen.
10.6 and prior Releases
CONCSUB applsys/<password> SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ABORT
The ICM can also be started for within the applications and from the command line. From the applications do the following:
In Char Mode (under SYSADMIN) go to: \Nav Con Man Admin
In GUI (under SYSADMIN) go to: Concurrent Manager Administrator
- You need to query the concurrent manager record, by doing a "Query Run".
- In the Control column either type 'Activate' or do a QuickPick and then save the screen.
10.6 and prior Releases
startmgr sysmgr=applsys/<password> mgrname='SID'
mgrname is if you want to name your concurrent managers.
If this option is chosen then the ICM logfile will now be the %lt;name%gt;mgrname.mgr.
diag - will create a more verbose logfile.
This option is normally used to troubleshoot ICM problems.
Other parameters like sleep, pmon, quesiz are described on pg. 8-18 of the System Administration Reference Manual Release 10.
Internal Concurrent Manager
There are three parameters that affect the performance of the ICM. These parameters are sleep time, pmon cycle, and queue size.
First, Sleep time. This default is 60 seconds. This is the number of seconds that the internal concurrent manager waits between times it looks for new concurrent requests.
Second, PMON. This is the number of sleep cycles the ICM waits between time it checks for failed concurrent managers. The default is 20.
Finally, Queue size. This is the number of PMON cycles that the ICM waits between times it checks for new or disabled concurrent managers. The queue size default is 1.
The above parameters can be set during startup of the ICM. The parameters pmon and quesiz can be accomplish if the user does a 'Verify' on the ICM within applications. The path is \Nav Con Man Admin
Product Managers
The sleep time for the product managers are set to the default of 60 seconds. This is the number of seconds that the manager waits between checking the list of pending concurrent requests. The sleep parameter is modified in the 'Work Shifts' zone of the 'Define Concurrent Manager' screen. \Nav Con Man Def
The last tuning parameter is called 'Cache Size'. The Cache Size is the number of concurrent requests that the manager picks up from the FND_CONCURRENT_REQUST table when the manager wakes up. For customer defined managers the Cache Size is either blank or set to 0. This number should be set to twice the number of target processes. If the target process is set to three (\N C M A) then the Cache Size should be six.
The reasoning is, if the manager has three target processes it could run three jobs. But if it only picks up one job every 60 seconds then you are losing valuable processing time. If it picks up six jobs then the managers will continuously be processing. This parameter can ONLY be set though the GUI. It can not be set from within character mode.
Install Issues
Passwords
10.7 the APPLSYS and APPS passwords MUST be the same. The user can change the default passwords, but both passwords MUST be the same. The problem is functionality will be lost if the passwords are different. You will be able to start the concurrent managers, but other items (i.e. adadmin, adpatch) will not work correctly. If the user decides to change the passwords they MUST follow the steps in GSX 1009719.6 - Resetting Applications Passwords (GSX is Oracles Global Solution Exchange)
Demo Database
When installing the 10.7 demo database the user must be aware that they might need to update a couple of tables for the concurrent managers to come up properly.
The reasoning is these are compressed .dbf files and the data that is contained within them is from another system. The following tables need to be updated:
(This is how we cleanup the FND tables)
truncate table FND_CONCURRENT_REQUESTS
truncate table FND_CONCURRENT_PROCESSES
After the above have been updated start the ICM in the diagnostics state. This will give you a more verbose logfile.
10.7
startmgr sysmgr=apps/<password> diag=y
10.6 and prior
startmgr sysmgr=applsys/<password> diag=y
When the managers come up look at the logfile 'std.mgr'. On the first page you should see (if you used diagnostics):
Process Monitor (PMON) Method is LOCK
If you do NOT see LOCK do the following. It is always a good idea to check with WorldWide Support before changing the PMON method.
- Shutdown the ICM (wait until it comes down)
- Go to $FND_TOP/sql/afimpmon.sql
- Run the above script as applsys/<password>
- Type the word 'dual' when it asked.
- Type 'LOCK'
The PMON method gives the customer three choices and they are:
OS = The internal concurrent manager determines whether or not a process is still running by sending a kill -0 to the process. This is merely a test to see if the process is still here. (THIS METHOD IS NO LONGER USED.)
RDBMS = The internal concurrent manager queries against v$process to retrieve the session id (the one that started the request) and the operating system process id. The reason it needs to query against both columns and not just the OS process id (pid) is because if there are concurrent managers running on two different clients against the same RDBMS then two requests could have the same OS pid.
LOCK = The internal concurrent manager tries to get a lock on the process it is monitoring. The name of the lock is determined by a sequence (which is the id of the individual managers (standard, etc)) and the program in question. Once the manager is able to get the lock, then it knows the process is no longer running. Two installs in one RDBMS share the lock method because if the two managers get the same sequence id (one in each instance) then the second manager will never be able to start because the ICM will always think its already running.
(THIS IS THE RECOMMENDED METHOD FROM 10.6 ON.) Caution: If using DCP, you can only have one system using LOCK. Others must be set to rdbms.
If you are still getting errors, check if you have any invalid objects in your database that start with FND_CON% owned by APPS or APPLSYS. If you have invalid objects recompile them, they could be causing the problem. If the packages will NOT recompile the packages will need to be recreated. Please contact support to help find and recreate the packages.
Fresh Install or Upgrade
With a fresh install you will not have to cleanup the concurrent manager tables (they have just been installed). Once you have tried to start the ICM go to the logfiles and check for errors. It is possible to find the following:
If you find the following -- Please note the following are NOT all of the possible errors you could receive - For a more comprehensive list of errors please see GSX 1011248.6 What are the common concurrent manager issues?:
If the manager fails to start please go to that section:
USDSOP failed during spawn of <Product>LIBR process
The above normally happens when an application is installed as shared. The seed data has been placed into FND_CONCURRENT_QUEUES and there is no executable on the file system. This is because the install has enabled that manager and the ICM cannot find the corresponding executable. The best example is if '<Product>LIBR' is getting the USDSOP error and <Product> was not in installed or it is installed as shared.
The fix for this problem is to set the target processes to 0 in 'Define Concurrent Managers' screen.
In Char Mode (under SYSADMIN) go to: \Nav Con Man Define
- Go to the last zone on the page and pick 'Work Shifts'
- Under target processes place a 0
- Save the Screen
- Go to \Nav Con Man Admin
- On the ICM tab over to the 'Control field' and do a 'Verify'
- Select 'Work Shifts'
- Place a 0 in 'Processes'
- Save the Screen
- Go to Concurrent Manager Admin
- Select 'Verify'
This is a generic error which may be followed by an Oracle error or:
APP-01054 - Distributed concurrent manager enabled, lock PMON method requested
APP-01114 - AFPCAL received failure code while running FNDCPMGR
Solution: Make certain that $APPLDCP is set to off or, if using distributed concurrent processing, ensure the PMON method is LOCK. (Only on primary node. Other nodes should use rdbms.)
Migration
During a migration you may find the same type of problems as in the DEMO install. The reasoning is, you are moving from one platform to another. If you have not moved your file system the ICM may not come up. The fix may be as simple as cleaning up the concurrent manager tables. It is possible that the concurrent manager is looking for files that were on the other system.
Start Up of Concurrent Manager Failed
Now we are getting to point where users might be breathing down your neck. You need to ask a few basic questions:
- Has anything changed either on your system or in the applications??
- What are the errors your receiving??
- APP-1167
- APP-1114
ps -ef | grep FNDLIBR
Are the number of FNDLIBR processes equal to 1 + the number of target processes from within your applications?
Please Note: If you have more than set of applications running on the system then you WILL get more back from your grep. But, if these are your only applications then you should not get anything but the grep back.
If the managers are not down submit a concurrent request to bring them down:
CONCSUB applsys/<password> SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ABORT
If this request only goes to pending never runs then we will need to kill the concurrent managers from the command line.
The Concurrent Managers are Down
1) Let's go to the logfiles. Remember they are in either $APPLCSF/$APPLLOG or $FND_TOP/$APPLLOG.
2) Rename the logfile. The reasoning here is, the ICM only appends to the log and it is much easier to start with a fresh log.
- Lets reset the tables in sqlplus.
- Logon as applsys/<password>
- You should get 0 count. If you do get a count back then delete them.
- You should get 0 count. If you do get a count back then delete them.
- Delete from FND_CONCURRENT_REQUESTS where status_code = 'C';
- Lets update FND_CONCURRENT_QUEUES update FND_CONCURRENT_QUEUES set running_processes=0, max_processes=0;
- The next one will remove all of entries in the table.
- Go back to the operating system (O/S)
- From the O/S lets start the concurrent managers with diagnostics mode turned on:
startmgr sysmgr=apps/<password> diag=y
10.6 or prior
startmgr sysmgr=applsys/<password> diag=y
- Now lets check the logfile for any errors. If you have any errors and they can range from USDSOP to APP-????. We need to address those.
- Now lets see if the managers have come up. This can be done from within the applications
In Char: \Nav Con Man Admin
- If the managers have not come up and you do not have any errors lets check for invalid objects.
- In sqlplus as system/<password>
- If the number comes back and is less than 10 then we need to know if any of then start with 'FND_'. If so then lets compile them and try to restart the ICM again.
- If the number comes back and is greater than 10 and less than 100. This can still be alright. Again we need to find out if any of them start with 'FND_'. If so then lets compile them and try to start the ICM again.
- If the number is greater than 100 we need to recompile the objects. Once that is complete then we can try the ICM again.
- What is the PMON method? If you've started the ICM with diagnostics then you will be able to tell what method the PMON method is set to. If the ICM is up, you can also check the PMON method by running afimchk.sql.
If you do NOT see LOCK do the following. It is always a good isea to check with WorldWide Support before changing the PMON method.
- Shutdown the ICM (wait until it comes down)
- Go to $FND_TOP/sql/afimpmon.sql
- Run the above script as applsys/<password>
- Type the word 'dual' when it asked.
- Type 'LOCK'
OS = The internal concurrent manager determines whether or not a process is still running by sending a kill -0 to the process. This is merely a test to see if the process is still there. (THIS METHOD IS NO LONGER USED.)
RDBMS = The internal concurrent manager queries against v$process to retrieve the session id (the one that started the request) and the operating system process id. The reason it needs to query against both columns and not just the OS pid is because if there are concurrent managers running on two different clients against the same RDBMS then two requests could have the same OS pid.
LOCK = The internal concurrent manager tries to get a lock on the process it is monitoring. The name of the lock is determined by a sequence (which is the id of the individual managers (standard, etc)) and the program in question. Once the manager is able to get the lock, then it knows the process is no longer running.
Two installs in one RDBMS share the lock method
because if the two managers get the same sequence id (one in each
instance) then the second manager will never be able to start
because the ICM will always think its already running.
(THIS IS THE RECOMMENDED METHOD FROM 10.6 ON.)
10.7 NT Concurrent Managers:
There are a few interesting issues with concurrent managers on NT. The concurrent managers are services on the NT. They have to be added and removed with the following commands.
Note: When adding the service you MUST type in the UserID and password. It is does NOT default. If you do NOT type in the password the you will get an ORA-1017 error in the event viewer log.
Normally the <APPL_CONFIG> will be either APPLSYS or APPDEMO. One point, only the user who added the service will be allowed to remove the service. After the service has been added then it needs to be started from 'Services' in the 'Control Panel'.
Errors when Starting the Managers on NT
Now if your like most people, at some point you'll get an error when starting the Concurrent Managers. The normal generic error from the 'Services' goes something like this:
Could not start the OracleConcMgr<APPL_COMFIG> service on 'Machine Name'
Error 2140: An internal Windows NT error occurred.
(The above error doesn't say a whole lot - to get a better error message go to: Start - Programs - Administrative Tools - Event Viewer Go to Log and Click on Application)
Find the Source of OracleConcMgr<APPL_CONFIG>
The Event Log Says:
The Description for Event ID (0) in Source [OracleConcMgr<APPL_CONFIG>] could not be found. It contains the following insertion string(s): Cannot connect to database. Sid=ORACLE_SID ORA error 12154.
The fix:
Add a loop-back in your tnsnames with the ORACLE_SID as the alias.
ANOTHER ERROR:
At the command line:
Service not started. Check the event log for more information.
Exit Code: 0
Service Specific Exit Code: 0
The Event Log Says:
The Description for Event ID (0) in Source [OracleConcMgr<APPL_CONFIG>] could not be found. It contains the following insertion string(s): OracleConcMgr<APPL_CONFIG> error 3: The system cannot find the path specified.
Cannot delete Concurrent Manager logfile.
\%FND_TOP%\logdemo\CM_LOG. Please check that these are no processes running.
The Fix:
Check to ensure the directory 'LOGDEMO' is in FND_TOP. On the NT even if APPLCSF is set, the concurrent manager log files will still write to FND_TOP/log.
ANOTHER ERROR:
At the command line:
Service not started. Check the event log for more information.
Exit Code: 0
Service Specific Exit Code: 0
The Event Log Says:
The Description for Event ID (0) in Source [OracleConcMgr<APPL_CONFIG>] could not be found. It contains the following insertion string(s): Cannot connect to database SID = <ORACLE_SID> ORA error 1017.
The Fix:
One of the following were typed incorrectly:
APPS schema's UserID:
APPS schema's password:
LAST ERROR I'LL LIST:
At the command line:
Failed to start Service OracleConcMgr<APPL_CONFIG>
Error: 1069 The service did not start due to a logon failure
The Event Log Says:
No event log for this error
The Fix:
The 'Windows NT user account password' was typed incorrectly:
Remove the service and add it back.
The following tables are the most important tables for the concurrent managers. These are not the only tables that have to do with the concurrent managers.
FND_CONCURRENT_PROCESSES
Stores information about concurrent managers. Each row includes values that identify the ORACLE process, the operating system process, and the concurrent manager (QUEUE_APPLICATION_ID and CONCURRENT_QUEUE_ID). You need one row for each instance of a running concurrent manager (each process), as well as one row for the Internal Concurrent Manager. AOL uses this table to keep a history of concurrent managers. You should never update this table manually.
Purge Concurrent Requests and /or Managers Data program to delete history information periodically.
FND_CONCURRENT_REQUESTS
Stores information about individual concurrent request. Each row includes values that identify the particular request and its parameters, such as who submitted it, the request type, whether the request should run sequentially with other requests in the same logical database (SINGLE_THREAD_FLAG), whether the request is on hold (HOLD_FLAG), whether to display the request in the View Requests form for the request submitter to review, and what status a phase the concurrent request is in. Each row also includes values that identify the concurrent program, its execution and argument methods, and whether the program is constrained (QUEUE_METHOD_CODE). Each row also includes flags that indicate the requests, priority related to other requests, as well as values that specify how the concurrent manager should print program output, if any.
ARGUMENT1 through ARGUMENT25 contain any arguments the application passes to the concurrent program. If the concurrent program needs more that 25 arguments to run, the first 25 arguments are stored in this table, ARGUMENT26 through ARGUMENT100 are stored in FND_CONC_REQUEST_ARGUMENTS. ARGUMENT_TEXT contains the concatenation of concurrent requests arguments and COMPLETION_TEXT contains a message about how the request completed. The row also contains dates that the request was submitted, requested to start and actually run. REQ_INFORMATINO is used with reports sets to remember the status of the request between runs. When the request is set to use automatic resubmission, RESUBMITTED is a flag to indicate whether the request has been resubmitted or not.
RESUBMIT_INTERVAL_TYPE_CODE specifies whether to start interval count down from the requested start time or the completion of the request. RESUBMIT_INTERVAL_TYPE_CODE indicates whether interval unit is in Days, Hours, Minutes, or Months. RESUBMIT_TIME set the time of the day to rerun the concurrent request. RESUBMIT_INTERVAL indicates the number of units of time when the identical request will be resubmitted. RESUBMIT_END_DATE is the date the request stops resubmitting itself. IS_SUB_REQUEST is a flag that identifies a child request and HAS_SUB_REQUEST is a flag that identifies a parent request. Each child request also needs to have values in PARENT_REQUEST_ID to show what parent request submitted the child request and PRIORITY_REQUEST_ID to tell what priority the parent request has and what priority the child request should have. You need one row for each concurrent request. Though you should occasionally delete from this table, you should not modify any of its data.
FND_CONCURRENT_QUEUES
Stores information about concurrent managers. Each row includes the name and description of a concurrent manager, as well as values that identify the program library attached to the manager. CACHE_SIZE contains the buffer size (how many requests a concurrent manager should "remember" each time it checks the list of waiting requests), and MAX_PROCESSES determines the maximum number of concurrent requests a manager can run at a time (depends on work shifts). Then manager process automatically updates RUNNING_PROCESSES during start up and shut down time. Each row also includes the time in seconds for the concurrent manager to wait before checking for pending concurrent requests, and a code to activate, deactivate, or reset the manager. You need one row for each concurrent manager defined at your site, with a minimum of one row. AOL uses this information to activate concurrent managers.
No comments:
Post a Comment