MQSeriesAdapter Exception: javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed.

After changing the configuration of the MQSeriesAdapter I found below exceptions in the logs:

[2019-09-13T13:18:35.551+02:00] [MS_OSB1] [ERROR] [] [oracle.soa.adapter.mq.inbound] [tid: JCA-work-instance:MQSeries Adapter-44] [userId: <WLS Kernel>] [ecid: 8483ccd4-5c52-41e7-bec5-bf8eeb60f49b-002ba627,0] [partition-name: DOMAIN] [tenant-name: GLOBAL]  [endpoint_name: Service_v1/Proxy/ServiceMQAS] [QueueProcessor] Exception caught in inbound cycle[[ javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to perform another lookup of the Connection Factory eis/MQ/MQSeriesAdapter from JNDI and get a new Connection Handle.

        at weblogic.connector.outbound.ConnectionManagerImpl.checkIfPoolIsValid(ConnectionManagerImpl.java:479)

        at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:115)

        at oracle.tip.adapter.mq.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:120)

        at oracle.tip.adapter.mq.inbound.BaseQueueHandler.getNewConnection(BaseQueueHandler.java:76)

        at oracle.tip.adapter.mq.inbound.QueueProcessor.run(QueueProcessor.java:238)

        at oracle.tip.adapter.sa.impl.fw.jca.work.WorkerJob.go(WorkerJob.java:53)

        at oracle.tip.adapter.sa.impl.fw.common.ThreadPool.run(ThreadPool.java:281)

        at java.lang.Thread.run(Thread.java:748)

Apparently the changes has caused the connections to the queue manager to become stale. The solution was simple. I had to restart the WebLogic server in order to make it work again!

JMS proxy service exception: Validation of RefValueStore _static/Ref_Store/ProxyService - JMS Transport.InboundJMSProxy

I have only started working with JMS queues recently and find that problem solving around JMS proxy's can be so time consuming. I recall my previous blog.
Last week I had another JMS proxy problem and couldn't find much info on the error I was getting, so hope this will help you out :)

After rework done on a JMS proxy service where I tried to move it to another project I noticed messages weren't read from the queue anymore. Redeploying didn't work and when I tried to delete the project for a clean deployment I got the following error message within the console: Validation of RefValueStore _static/Ref_Store/ProxyService - JMS Transport.InboundJMSProxy. : RefValueStore _static/
Ref_Store/ProxyService - JMS Transport.InboundJMSProxy. does not exist failed.

It seemed that the JMS proxy was unable to find its related MDB. Every time a JMS proxy service is deployed an MDB is created that handles the connection with the JMS queue. See also my previous blog for more details. In this case OSB was unable to verify the MDB as it didn't exist anymore. You can check the deployments in the WebLogic Server, MDB deployments have the prefix SB_JMS_Proxy_*.

So I was basically stuck with a corrupt OSB service and couldn't remove or redeploy it to fix the problem. I figured I had to redeploy the service so the MDB is recreated and in order to do that I had to delete the service manually through the server. I did as follows to resolve the problem:

-Find the project or folder of your service in <OSB Domain>/osb/configfwk/core and rename the folder.
-Stop the AdminServer.
-Backup and clear cache, log and tmp folder of AdminServer.
-Start the AdminServer. When you now check the OSB console you will see the service is gone.
-In your IDE change the name of the project or folder of your service and deploy to OSB server.
-A new MDB should be created. Verify this in the deployments list on the WebLogic Server.
-Rename your project or folder to its original name.

These steps helped me to resolve the problem and messages were read from the queue again.

Working with an SOA MDS: using abstract WSDL's

The WSDL plays an essential role in defining a web service. Besides describing the structure of the SOAP message, it often contains additional implementation details about the location of the service by having the endpoint described in the service tag. This is what we call a concrete WSDL and it may look as follows:

When we populate our MDS we want it to contain abstract WSDL’s that do not contain these implementation details. This way we take away the direct dependencies between services. It means that the actual referenced services don’t need to be deployed and active in order for our SOA composite to be compiled. It also means we keep our MDS clean in the sense that it does not have any references to other environments.

The way of working with abstract WSDL’s in jDeveloper differs slightly compared to concrete WSDL’s. When creating a reference to a service, you would typically select the WSDL needed from the MDS, as below:

In contrary to using a concrete WSDL you will now get the following warning to notify you the WSDL doesn’t contain any bindings. This is perfectly fine and we can just continue by clicking ‘Yes’.

If you now check the composite.xml, you will notice that no binding details were provided in the binding.ws tag.

We need to provide them manually by looking up the concrete WSDL within the development environment. The format is as follows for the port details:
<Namespace service>#wsdl.endpoint(Service name/Port name)

In our case it becomes: http://www.stellent.com/FolderInfo/#wsdl.endpoint(FolderInfo/FolderInfoSoap)

In the location attribute we provide the endpoint. Altogether your service reference should look like this:

When we create a deployment for installation on a different environment we will create a deployment plan which contains the endpoints of the target environment. This will overwrite the endpoint we provided in the location attribute of the composite.xml.

Exception after redeploying JMS proxy service on OSB 12c: Failure while processing an incoming message for endpoint ProxyService EventQueue/ProxyServices/PS_EventQueue due to service configuration in flux

So I had to redeploy my JMS proxy on OSB and I noticed messages weren’t getting through anymore. When I checked the logs I saw the following exception when the proxy tried to process the messages:

[2018-05-04T15:42:19.696+02:00] [osb_server1] [ERROR] [OSB-381502] [oracle.osb.transports.main.jmstransport] [tid: [ACTIVE].ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 005Qq5tI3tl4QtD5vBt1iX0003Q30003tS,1:1] [APP: SB_JMS_Proxy_a470f11.N7ff2315c.0.162e3b1fee2.N8000] [partition-name: DOMAIN] [tenant-name: GLOBAL] Exception in JmsInboundMDB.onMessage: com.bea.wli.sb.transports.TransportException: Failure while processing an incoming message for endpoint ProxyService EventQueue/ProxyServices/PS_EventQueue due to service configuration in flux[[

com.bea.wli.sb.transports.TransportException: Failure while processing an incoming message for endpoint ProxyService EventQueue/ProxyServices/PS_EventQueue due to service configuration in flux

        at com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:121)

        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)

        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)

        at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:297)

        at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:5107)

        at weblogic.jms.client.JMSSession.execute(JMSSession.java:4775)

        at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:4170

        at weblogic.jms.client.JMSSession.access$000(JMSSession.java:127)

        at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5627)

        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:666)

        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)

        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)

        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)

        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)

        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)

This problem was caused, so I learned later, by the fact that messages were still being processed while I was redeploying my proxy. For every redeployment the JMS proxy will update the Message-Driven Bean (MDB) that will take care of the message consumption. This MDB is deployed on the WebLogic Server.  It could take some time for this MDB to be updated and start consuming messages again and get in sync with the pipeline runtime objects. This will cause the service to be “in flux”.

To solve the problem I had tried the obvious first: redeploying the service, restarting the server and the JMS queue. None of this worked because the problem was in the MDB, so I found out. Eventually I did the following to resolve my problem:

• Undeploy your JMS service in OSB.
• Select your JMS Server via WebLogic Server Console: Services > Messaging > JMS Servers  and go to tab “Control”. Pause Production and Consumption.
• Look for the config.xml file located in your <OSB Domain>/config folder. In this file, look for the app-deployment that represents your MDB. You can identify it by checking the APP ID in the exception. I marked it yellow in the exception above. It should correspond with the ID in the name tag. Remove the app-deployment part.
  <app-deployment>
    <name>SB_JMS_Proxy_a470f11.N7ff2315c.0.162e3b1fee2.N8000</name>
      <target>osb_cluster</target>
      <module-type>ear</module-type>
      <source-path>sbgen/jms/SB_JMS_Proxy_a470f11.N7ff2315c.0.162e3b1fee2.N8000.ear</source-path>
      <plan-path>sbgen/jms/plan/SB_JMS_Proxy_a470f11.N7ff2315c.0.162e3b1fee2.N8000_Plan.xml</plan-path>
      <security-dd-model>DDOnly</security-dd-model>
      <staging-mode>stage</staging-mode>
      <plan-staging-mode xsi:nil="true"></plan-staging-mode>
      <cache-in-app-directory>false</cache-in-app-directory>
    </app-deployment>
• Go to your Deployments in the OSB WebLogic Server Console.
• In your list you should see the MDB’s for your JMS proxies. The names start with SB_JMS_Proxy_xxxx. If you’re working with OSB 11g the prefix is _ALSB_xxxx.
  In my case I could identify the MDB quickly as health was set to “Warning”. Look up the MDB and remove it. I’m not sure if a server restart is necessary but I restarted to be sure.
  

  
  

  
  
• Redeploy your JMS service to OSB, resume actions for your JMS Server and your proxy service should be processing the messages again.
  
  Please note that I advise against executing these steps in a production environment as data might get lost!
  
  My lessons learned? Next time I need to redeploy my JMS proxy service I will make sure the JMS Server is paused to prevent the in flux error.
  
  Did this blog help resolve your problem? Let me know in a comment!