Jump to: navigation, search

Difference between revisions of "WTP/HTTP Connection Tracer Diagnostic Utility"

< WTP
(initial transfered draft)
 
m
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
<h1>HTTP Connection Tracer Diagnostic Utility</h1>
+
= HTTP Connection Tracer Diagnostic Utility =
<h2>The Problem</h2>
+
<p>There are times that code is seen to go to the internet to fetch something, and its not often clear why or what code is responsible. Sometimes it is necessary for code to fetch something remotely, but often it is best to keep a local cached copy, use and XML Catalog, etc., to avoid excessive internet traffic.</p>
+
<h2>The solution (to help diagnosis excessive access)</h2>
+
<p>There is an "HTTP Connection Tracer" tool to help detect and track when code requests an HTTP connection, so that excessive requests can be detected early, and more easily fixed.
+
</p>
+
  
In addition to logging request, the tool does allow a "remapping" of an HTTP request to a local file. This can help work around immediate problems, but of course, is not a long term solution.
+
== The Problem ==
  
<p>Please address any questions, comments or suggestions in <a
+
There are times that code is seen to go to the internet to fetch something, and its not often clear why or what code is responsible. Sometimes it is necessary for code to fetch something remotely, but often it is best to keep a local cached copy, use and XML Catalog, etc., to avoid excessive internet traffic.
    href="https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Web%20Tools"
+
    target="_top">bugs</a> or via the <a
+
    href="https://dev.eclipse.org/mailman/listinfo/wtp-dev"
+
    target="_top">wtp-dev@eclipse.org</a> mailing list.</p>
+
  
<h3>Installation</h3>
+
== The solution (to help diagnosis excessive access)  ==
<p>Download the <a href="http://www.eclipse.org/webtools/development/httptracerutility/httphandler.jar">HTTP connection tracer diagnostic utility</a> jar file.</p>
+
 
<p>The jar file must be added to the JRE's lib/ext directory. This must be done for each JRE that you're
+
There is an "HTTP Connection Tracer" tool to help detect and track when code requests an HTTP connection, so that excessive requests can be detected early, and more easily fixed.
using in your environment. So if you're developing your code with eclipse and deploying to a server (and you want to get
+
 
trace information for both) you'll need to make the JAR available for both the eclipse JRE and the server's JRE.</p>
+
In addition to logging request, the tool does allow a "remapping" of an HTTP request to a local file. This can help work around immediate problems but, of course, is not a long term solution.
<h3>Configuration</h3>
+
 
<p>When you launch eclipse or run your sever, you'll need to specify some vmargs to activate the "HTTP Connection
+
Please address any questions, comments or suggestions in [https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Web%20Tools bugs] or via the [https://dev.eclipse.org/mailman/listinfo/wtp-dev wtp-dev@eclipse.org] mailing list.
Tracer" tool. Here's an example of how you'd do this on the command line for Eclipse.</p>
+
 
 +
Note: this utility only detects HTTP requests that are made through the Java VM APIs. There are other frameworks, such as org.apache.commons.httpclient and ECF that provide other APIs (this is why, for example, p2 related http requests would not normally show up in this utility). <br>
 +
 
 +
Note: this utility was written by Craig Salter and provided to Ecliple open source via {{bug|109788}}, in 2006. Thanks go to him. It was (much) later renamed to conform "eclipse.org" namespace, and put into Web Tools CVS (/cvsroot/webtools), in webtools.releng/tests/org.eclipse.wtp.releng.httphandler. This jar is not built during any of our regular builds, but was built from a developers workspace, and placed on downloads server, and /shared/webtools/apps, so whenever a new VM is used, it should be easy to find the built version to put in the jre/lib/ext directory. <br>
 +
 
 +
=== Installation  ===
 +
 
 +
Download the [http://www.eclipse.org/webtools/development/httptracerutility/org.eclipse.wtp.releng.www.protocol.http_1.1.0.201109252208.jar HTTP connection tracer diagnostic utility] jar file.  
 +
 
 +
The jar file must be added to the JRE's lib/ext directory. This must be done for each JRE that you're using in your environment. So if you're developing your code with eclipse and deploying to a server (and you want to get trace information for both) you'll need to make the JAR available for both the eclipse JRE and the server's JRE.
 +
 
 +
Note: even though the current jar is an OSGi bundle, with a normal OSGi bundle version, when it is added to the lib/ext directory, it is treated as a normal jar ... so, there should be only one version of the jar there in lib/ext directory. There is no OSGi loading, that would guarantee the highest version is loaded.
 +
 
 +
=== Configuration ===
 +
 
 +
When you launch eclipse or run your sever, you'll need to specify some vmargs to activate the "HTTP Connection Tracer" tool. Here's an example of how you'd do this on the command line for Eclipse.  
 +
<pre> eclipse.exe -vmargs -Djava.protocol.handler.pkgs=org.eclipse.wtp.releng.www.protocol -DurlMapLocation=C:\map.txt -DurlLogLocation=C:\log.txt
 +
</pre>  
 +
Here's a description of what each argument does ....
 +
 
 +
{| border="1" cellpadding="5"
 +
|- valign="top"
 +
! name
 +
! required or example value
 +
! description
 +
|- valign="top"
 +
| java.protocol.handler.pkgs
 +
| org.eclipse.wtp.releng.www.protocol
 +
| Tells the JVM to utilize the "HTTP Connection Tracer" tool. (required, must match actual package name in jar. Search the internet for 'java.protocol.handler.pkgs' for information about this general scheme of overriding default handlers.)
 +
|- valign="top"
 +
| urlMapLocation
 +
| C:\map.txt
 +
| Specifies the location of a 'properties' file where URL mapping information can be provided. Note that if no map location is specified then URL redirection never occurs, though logging still will.
 +
|- valign="top"
 +
| urlLogLocation
 +
| C:\log.txt
 +
| Specifies the location of a log file where logging information will be written. Note that if no log location is specified the logging information will be written to System.out. [ISSUE: is the following sentence still true? The log might always be created now?] Note also that if no HTTP requests are made, then no log or output is produced. If there is no output, you may want to confirm your setup by running a small test program that explicitly does a URL connection. See {{bug|216524}} for an example.
 +
|}
 +
 
 +
==== The Map File  ====
 +
 
 +
Here's an example of what the map file looks like....
 +
<blockquote>http://www.hello.com/bar.xsd =file:///D:/my-schema-cache/bar.xsd<br> http://www.ibm.com/foo.xsd=file:///D:/my-schema-cache/foo.xsd</blockquote>
 +
Notice its just a list of mapping pairs (separated by an '=' sign) that specifies how a web address should be redirected to a local file system address. By editing this file, a customer can workaround nagging 'going off to the internet problems' until a proper fix is delivered.
 +
 
 +
==== The Log File  ====
 +
 
 +
Each time the JVM attempts to create an HTTP connection, an entry is added to the log file. Below is an example log file entry. It consists of three interesting pieces of information...
 +
<blockquote>URL request - provides the URL address for the connection requested<br> URL mapped - provides the 'mapped' address (as specified in the 'map.txt' file) to help the customer see that the URL has been successfully redirected<br> STACK TRACE - dumps of stack to help the support and development teams understand the code paths involved in creating the connection request</blockquote>
 +
Note that the presence of an entry in the log does not necessarily imply an defect in code. Some attempts to create an HTTP connections are expected. So I'd encourage you to scrutinize the entries in the log file. At the very least the logs will provide you some useful stack trace information that will help developers understand the nature of the HTTP connections.
 +
 
 +
=== Example Output  ===
 
<pre>
 
<pre>
  eclipse.exe -vmargs -Djava.protocol.handler.pkgs=org.eclipse.wtp.releng.www.protocol -DurlMapLocation=C:\map.txt -DurlLogLocation=C:\log.txt
+
--------------------------------------
 +
HTTP Connection Tracer is enabled
 +
For more information about HTTP Connection Tracer, see http://www.eclipse.org/webtools/development/httptracerutility/index.php
 +
urlLogLocation: /home/shared/webtools/tests/wtp4x-R3.4.0-I/results/httplogstest/outgoinghttplog-org.eclipse.jst.j2ee.core.tests.log
 +
urlMapLocation: (no location provided)
 +
---------------------------------------------
 +
URL requested : http://www.ibm.com/webservices/dtd/j2ee_jaxrpc_mapping_1_0.dtd
 +
URL mapped    : (not mapped)
 +
STACK TRACE  :
 +
java.lang.Exception: stack at time of request
 +
at org.eclipse.wtp.releng.www.protocol.http.Handler.openConnection(Handler.java:153)
 +
at java.net.URL.openConnection(URL.java:957)
 +
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
 +
at org.apache.xerces.impl.XMLEntityManager.startEntity(Unknown Source)
 +
at org.apache.xerces.impl.XMLEntityManager.startDTDEntity(Unknown Source)
 +
at org.apache.xerces.impl.XMLDTDScannerImpl.setInputSource(Unknown Source)
 +
at org.apache.xerces.impl.XMLDocumentScannerImpl$DTDDispatcher.dispatch(Unknown Source)
 +
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
 +
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 +
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 +
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
 +
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
 +
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
 +
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
 +
at org.eclipse.wst.common.internal.emf.utilities.DOMUtilities.createNewDocument(DOMUtilities.java:545)
 +
at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.createDocument(EMF2DOMRenderer.java:156)
 +
at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.createDOMTree(EMF2DOMRenderer.java:127)
 +
at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.createDOMTreeIfNecessary(EMF2DOMRenderer.java:144)
 +
at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.doSave(EMF2DOMRenderer.java:90)
 +
at org.eclipse.wst.common.internal.emf.resource.TranslatorResourceImpl.doSave(TranslatorResourceImpl.java:190)
 +
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.save(ResourceImpl.java:1423)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.saveMofResource(SaveStrategyImpl.java:182)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectorySaveStrategyImpl.saveMofResource(DirectorySaveStrategyImpl.java:173)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.saveMofResource(SaveStrategyImpl.java:171)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.basicSaveMofResources(SaveStrategyImpl.java:218)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.saveMofResources(SaveStrategyImpl.java:211)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.save(SaveStrategyImpl.java:99)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.ArchiveImpl.save(ArchiveImpl.java:1091)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.ModuleFileImpl.save(ModuleFileImpl.java:165)
 +
at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.ArchiveImpl.extractTo(ArchiveImpl.java:479)
 +
at org.eclipse.jst.j2ee.archive.emftests.WebServicesEMFTest.testJaxRPCMapPopulation(WebServicesEMFTest.java:124)
 +
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 +
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
 +
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
 +
at java.lang.reflect.Method.invoke(Method.java:611)
 +
at junit.framework.TestCase.runTest(TestCase.java:168)
 +
at junit.framework.TestCase.runBare(TestCase.java:134)
 +
at junit.framework.TestResult$1.protect(TestResult.java:110)
 +
at junit.framework.TestResult.runProtected(TestResult.java:128)
 +
at junit.framework.TestResult.run(TestResult.java:113)
 +
at junit.framework.TestCase.run(TestCase.java:124)
 +
at junit.framework.TestSuite.runTest(TestSuite.java:243)
 +
at junit.framework.TestSuite.run(TestSuite.java:238)
 +
at junit.framework.TestSuite.runTest(TestSuite.java:243)
 +
at junit.framework.TestSuite.run(TestSuite.java:238)
 +
at junit.framework.TestSuite.runTest(TestSuite.java:243)
 +
at junit.framework.TestSuite.run(TestSuite.java:238)
 +
at junit.framework.TestSuite.runTest(TestSuite.java:243)
 +
at junit.framework.TestSuite.run(TestSuite.java:238)
 +
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:501)
 +
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:259)
 +
at org.eclipse.test.CoreTestApplication.runTests(CoreTestApplication.java:36)
 +
at org.eclipse.test.CoreTestApplication.run(CoreTestApplication.java:32)
 +
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 +
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
 +
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
 +
at java.lang.reflect.Method.invoke(Method.java:611)
 +
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
 +
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
 +
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
 +
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
 +
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
 +
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
 +
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 +
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
 +
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
 +
at java.lang.reflect.Method.invoke(Method.java:611)
 +
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:624)
 +
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:579)
 +
at org.eclipse.equinox.launcher.Main.run(Main.java:1431)
 +
at org.eclipse.equinox.launcher.Main.main(Main.java:1407)
 
</pre>
 
</pre>
<p>Here's a description of what each argument does ....</p>
+
 
<table border="1">
+
== The solution (for fixing code that makes excessive http requests) ==
    <tr>
+
 
        <th>name</th>
+
While there is no one solution for all situations where you want to avoid or minimize network access, here we will mention a few things to consider.  
        <th>example value</th>
+
 
        <th>description</th>
+
*Use an XML Catalog, with copies of resources on your local disk.  
    </tr>
+
*Set validation off, and set a null resolver.  
    <tr>
+
*Use a cache, so resources are at least only retrieved once, such as the cache in WTP!
        <td>java.protocol.handler.pkgs</td>
+
 
        <td>com.ibm.etools.www.protocol</td>
+
== How WTP utilizes this utility in JUnit tests  ==
        <td>Tells the JVM to utilize the "HTTP Connection Tracer" tool.</td>
+
 
    </tr>
+
When WTP runs its JUnit tests, it passes in the VM arguments to enable this tracing utility when the JUnits run. We "manually" copy over the jar file to the test JRE's lib/ext directory. We do not "fail" (currently) for any such logging, but provide the reports for committers to browse so they can see if there is anything suspicious logged during the running of their code. To find the logs, first look at the bottom of the JUnit summary page:
    <tr>
+
 
        <td>urlMapLocation</td>
+
[[Image:Httpsummary.png|frame|left]]
        <td>C:\map.txt</td>
+
 
        <td>Specifies the location of a 'properties' file where URL mapping information can be provided. Note that
+
The "http logs" link will show a list of logs captured, for each JUnit suite, a partial example image is below:
        if no map location is specified then URL redirection never occurs.</td>
+
 
    </tr>
+
[[Image:Httpreports.png|frame|left]]
    <tr>
+
 
        <td>urlLogLocation</td>
+
 
        <td>C:\log.txt</td>
+
One thing to look for is a log that is a lot larger than the others, as it might indicate excessive internet requests. Of course, they might all be normal, expected internet connections, but it takes someone familiar with the code and function to know.
        <td>Specifies the location of a log file where logging information will be written. Note that if no log
+
 
        location is specified the logging information will be written to System.out. Note also that if no HTTP requests
+
 
        are made, then no log or output is produced. If there is no output, you may want to confirm your setup by
+
 
        running a small test program that explicitly does a URL connection. See <a
+
[[Category:WTP_Build_Related| WTP_Build_Related]]
            href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=216524">bug 216524</a> for an example.</td>
+
    </tr>
+
</table>
+
<h4>The Map File</h4>
+
<p>Here's an example of what the map file looks like....</p>
+
<blockquote>http://www.hello.com/bar.xsd =file:///D:/my-schema-cache/bar.xsd<br />
+
http://www.ibm.com/foo.xsd=file:///D:/my-schema-cache/foo.xsd</blockquote>
+
<p>Notice its just a list of mapping pairs (separated by an '=' sign) that specifies how a web address should be
+
redirected to a local file system address. By editing this file, a customer can workaround nagging 'going off to the
+
internet problems' until a proper fix is delivered.</p>
+
<h4>The Log File</h4>
+
<p>Each time the JVM attempts to create an HTTP connection, an entry is added to the log file. Below I've shown an
+
example log file entry. It consists of three interesting pieces of information...</p>
+
<blockquote>URL request - provides the URL address for the connection<br />
+
URL mapped - provides the 'mapped' address (as specified in the 'map.txt' file) to help the customer see that the URL
+
has been succesfully redirected<br />
+
STACK TRACE - dumps of stack to help the support and development teams understand the code paths involved in creating
+
the connection request</blockquote>
+
<p>Note that the presense of an entry in the log does not neccessarily imply an defect in the product. Some attempts
+
to create an HTTP connections are expected. So I'd encourage you to scrutinize the entries in the log file. At the very
+
least the logs will provide you some useful stack trace information that will help developers understand the nature of
+
the HTTP connections.</p>
+
<h3>Example Output</h3>
+
<pre>
+
<code>
+
URL requested : http://www.ibm.com/foo.xsd
+
URL mapped : file:///D:/workspaces/corona-test/XMLExamples/substitutionGroup/Catalogue4.xsd
+
STACK TRACE :
+
java.lang.Exception: dumpTheStack
+
&nbsp;&nbsp;&nbsp;at com.ibm.etools.www.protocol.http.Handler.openConnection(Handler.java:156)  
+
&nbsp;&nbsp;&nbsp;at java.net.URL.openConnection(URL.java:924)
+
&nbsp;&nbsp;&nbsp;at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at javax.xml.parsers.SAXParser.parse(Unknown Source)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.xsd.util.XSDParser.parse(XSDParser.java:175)
+
&nbsp;&nbsp;&nbsp;at com.ibm.xsdeditor.internal.wizards.XSDSelectIncludeFileWizard.doLoadExternalModel(XSDSelectIncludeFileWizard.java:183)
+
&nbsp;&nbsp;&nbsp;at com.ibm.xsdeditor.internal.wizards.XSDSelectIncludeFileWizard$XSDURLPage.openExternalSchema(XSDSelectIncludeFileWizard.java:293)
+
&nbsp;&nbsp;&nbsp;at com.ibm.xsdeditor.internal.wizards.XSDSelectIncludeFileWizard$XSDURLPage.access$0(XSDSelectIncludeFileWizard.java:271)
+
&nbsp;&nbsp;&nbsp;at com.ibm.xsdeditor.internal.wizards.XSDSelectIncludeFileWizard$1.run(XSDSelectIncludeFileWizard.java:318)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:303)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:253)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.wizard.WizardDialog.run(WizardDialog.java:752)
+
&nbsp;&nbsp;&nbsp;at com.ibm.xsdeditor.internal.wizards.XSDSelectIncludeFileWizard.performFinish(XSDSelectIncludeFileWizard.java:130)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.wizard.WizardDialog.finishPressed(WizardDialog.java:609)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.wizard.WizardDialog.buttonPressed(WizardDialog.java:305)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.dialogs.Dialog$2.widgetSelected(Dialog.java:506)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:89)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java(Compiled Code))
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:796)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:2772)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java(Compiled Code))
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.window.Window.runEventLoop(Window.java:668)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.jface.window.Window.open(Window.java:648)
+
&nbsp;&nbsp;&nbsp;at com.ibm.xsdeditor.internal.properties.section.NamespaceAndSchemaLocationSection.widgetSelected(NamespaceAndSchemaLocationSection.java:159)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:89)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java(Compiled Code))
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:796)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:2772)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:2431)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:1377)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:1348)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:254)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:141)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.ui.internal.ide.IDEApplication.run(IDEApplication.java:96)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:335)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:273)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:129)
+
&nbsp;&nbsp;&nbsp;at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+
&nbsp;&nbsp;&nbsp;at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
+
&nbsp;&nbsp;&nbsp;at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
+
&nbsp;&nbsp;&nbsp;at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
+
&nbsp;&nbsp;&nbsp;at java.lang.reflect.Method.invoke(Method.java:391)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.core.launcher.Main.basicRun(Main.java:185)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.core.launcher.Main.run(Main.java:704)
+
&nbsp;&nbsp;&nbsp;at org.eclipse.core.launcher.Main.main(Main.java:688)
+
</code>
+
</pre>
+
<h2>The solution (for code)</h2>
+
<p>While there is no one solution for all situations where you want to avoid or minimize network access, here we will mention a few
+
things to consider.</p>
+
<ul>
+
    <li>Use an XML Catalog, with copies of resources on your local disk.</li>
+
    <li>Set validation off, and set a null resolver.</li>
+
    <li>Use a cache, so resources are at least only retrieved once, such as the cache in WTP!</li>
+
</ul>
+

Latest revision as of 11:12, 26 September 2011

HTTP Connection Tracer Diagnostic Utility

The Problem

There are times that code is seen to go to the internet to fetch something, and its not often clear why or what code is responsible. Sometimes it is necessary for code to fetch something remotely, but often it is best to keep a local cached copy, use and XML Catalog, etc., to avoid excessive internet traffic.

The solution (to help diagnosis excessive access)

There is an "HTTP Connection Tracer" tool to help detect and track when code requests an HTTP connection, so that excessive requests can be detected early, and more easily fixed.

In addition to logging request, the tool does allow a "remapping" of an HTTP request to a local file. This can help work around immediate problems but, of course, is not a long term solution.

Please address any questions, comments or suggestions in bugs or via the wtp-dev@eclipse.org mailing list.

Note: this utility only detects HTTP requests that are made through the Java VM APIs. There are other frameworks, such as org.apache.commons.httpclient and ECF that provide other APIs (this is why, for example, p2 related http requests would not normally show up in this utility).

Note: this utility was written by Craig Salter and provided to Ecliple open source via bug 109788, in 2006. Thanks go to him. It was (much) later renamed to conform "eclipse.org" namespace, and put into Web Tools CVS (/cvsroot/webtools), in webtools.releng/tests/org.eclipse.wtp.releng.httphandler. This jar is not built during any of our regular builds, but was built from a developers workspace, and placed on downloads server, and /shared/webtools/apps, so whenever a new VM is used, it should be easy to find the built version to put in the jre/lib/ext directory.

Installation

Download the HTTP connection tracer diagnostic utility jar file.

The jar file must be added to the JRE's lib/ext directory. This must be done for each JRE that you're using in your environment. So if you're developing your code with eclipse and deploying to a server (and you want to get trace information for both) you'll need to make the JAR available for both the eclipse JRE and the server's JRE.

Note: even though the current jar is an OSGi bundle, with a normal OSGi bundle version, when it is added to the lib/ext directory, it is treated as a normal jar ... so, there should be only one version of the jar there in lib/ext directory. There is no OSGi loading, that would guarantee the highest version is loaded.

Configuration

When you launch eclipse or run your sever, you'll need to specify some vmargs to activate the "HTTP Connection Tracer" tool. Here's an example of how you'd do this on the command line for Eclipse.

 eclipse.exe -vmargs -Djava.protocol.handler.pkgs=org.eclipse.wtp.releng.www.protocol -DurlMapLocation=C:\map.txt -DurlLogLocation=C:\log.txt

Here's a description of what each argument does ....

name required or example value description
java.protocol.handler.pkgs org.eclipse.wtp.releng.www.protocol Tells the JVM to utilize the "HTTP Connection Tracer" tool. (required, must match actual package name in jar. Search the internet for 'java.protocol.handler.pkgs' for information about this general scheme of overriding default handlers.)
urlMapLocation C:\map.txt Specifies the location of a 'properties' file where URL mapping information can be provided. Note that if no map location is specified then URL redirection never occurs, though logging still will.
urlLogLocation C:\log.txt Specifies the location of a log file where logging information will be written. Note that if no log location is specified the logging information will be written to System.out. [ISSUE: is the following sentence still true? The log might always be created now?] Note also that if no HTTP requests are made, then no log or output is produced. If there is no output, you may want to confirm your setup by running a small test program that explicitly does a URL connection. See bug 216524 for an example.

The Map File

Here's an example of what the map file looks like....

http://www.hello.com/bar.xsd =file:///D:/my-schema-cache/bar.xsd
http://www.ibm.com/foo.xsd=file:///D:/my-schema-cache/foo.xsd

Notice its just a list of mapping pairs (separated by an '=' sign) that specifies how a web address should be redirected to a local file system address. By editing this file, a customer can workaround nagging 'going off to the internet problems' until a proper fix is delivered.

The Log File

Each time the JVM attempts to create an HTTP connection, an entry is added to the log file. Below is an example log file entry. It consists of three interesting pieces of information...

URL request - provides the URL address for the connection requested
URL mapped - provides the 'mapped' address (as specified in the 'map.txt' file) to help the customer see that the URL has been successfully redirected
STACK TRACE - dumps of stack to help the support and development teams understand the code paths involved in creating the connection request

Note that the presence of an entry in the log does not necessarily imply an defect in code. Some attempts to create an HTTP connections are expected. So I'd encourage you to scrutinize the entries in the log file. At the very least the logs will provide you some useful stack trace information that will help developers understand the nature of the HTTP connections.

Example Output

--------------------------------------
HTTP Connection Tracer is enabled
For more information about HTTP Connection Tracer, see  http://www.eclipse.org/webtools/development/httptracerutility/index.php
urlLogLocation: /home/shared/webtools/tests/wtp4x-R3.4.0-I/results/httplogstest/outgoinghttplog-org.eclipse.jst.j2ee.core.tests.log
urlMapLocation: (no location provided)
---------------------------------------------
URL requested : http://www.ibm.com/webservices/dtd/j2ee_jaxrpc_mapping_1_0.dtd
URL mapped    : (not mapped)
STACK TRACE  :
java.lang.Exception: stack at time of request
	at org.eclipse.wtp.releng.www.protocol.http.Handler.openConnection(Handler.java:153)
	at java.net.URL.openConnection(URL.java:957)
	at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityManager.startEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityManager.startDTDEntity(Unknown Source)
	at org.apache.xerces.impl.XMLDTDScannerImpl.setInputSource(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentScannerImpl$DTDDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
	at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
	at org.eclipse.wst.common.internal.emf.utilities.DOMUtilities.createNewDocument(DOMUtilities.java:545)
	at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.createDocument(EMF2DOMRenderer.java:156)
	at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.createDOMTree(EMF2DOMRenderer.java:127)
	at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.createDOMTreeIfNecessary(EMF2DOMRenderer.java:144)
	at org.eclipse.wst.common.internal.emf.resource.EMF2DOMRenderer.doSave(EMF2DOMRenderer.java:90)
	at org.eclipse.wst.common.internal.emf.resource.TranslatorResourceImpl.doSave(TranslatorResourceImpl.java:190)
	at org.eclipse.emf.ecore.resource.impl.ResourceImpl.save(ResourceImpl.java:1423)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.saveMofResource(SaveStrategyImpl.java:182)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.DirectorySaveStrategyImpl.saveMofResource(DirectorySaveStrategyImpl.java:173)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.saveMofResource(SaveStrategyImpl.java:171)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.basicSaveMofResources(SaveStrategyImpl.java:218)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.saveMofResources(SaveStrategyImpl.java:211)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.strategy.SaveStrategyImpl.save(SaveStrategyImpl.java:99)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.ArchiveImpl.save(ArchiveImpl.java:1091)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.ModuleFileImpl.save(ModuleFileImpl.java:165)
	at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.ArchiveImpl.extractTo(ArchiveImpl.java:479)
	at org.eclipse.jst.j2ee.archive.emftests.WebServicesEMFTest.testJaxRPCMapPopulation(WebServicesEMFTest.java:124)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
	at java.lang.reflect.Method.invoke(Method.java:611)
	at junit.framework.TestCase.runTest(TestCase.java:168)
	at junit.framework.TestCase.runBare(TestCase.java:134)
	at junit.framework.TestResult$1.protect(TestResult.java:110)
	at junit.framework.TestResult.runProtected(TestResult.java:128)
	at junit.framework.TestResult.run(TestResult.java:113)
	at junit.framework.TestCase.run(TestCase.java:124)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:501)
	at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:259)
	at org.eclipse.test.CoreTestApplication.runTests(CoreTestApplication.java:36)
	at org.eclipse.test.CoreTestApplication.run(CoreTestApplication.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
	at java.lang.reflect.Method.invoke(Method.java:611)
	at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
	at java.lang.reflect.Method.invoke(Method.java:611)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:624)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:579)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1431)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1407)

The solution (for fixing code that makes excessive http requests)

While there is no one solution for all situations where you want to avoid or minimize network access, here we will mention a few things to consider.

  • Use an XML Catalog, with copies of resources on your local disk.
  • Set validation off, and set a null resolver.
  • Use a cache, so resources are at least only retrieved once, such as the cache in WTP!

How WTP utilizes this utility in JUnit tests

When WTP runs its JUnit tests, it passes in the VM arguments to enable this tracing utility when the JUnits run. We "manually" copy over the jar file to the test JRE's lib/ext directory. We do not "fail" (currently) for any such logging, but provide the reports for committers to browse so they can see if there is anything suspicious logged during the running of their code. To find the logs, first look at the bottom of the JUnit summary page:

Httpsummary.png

The "http logs" link will show a list of logs captured, for each JUnit suite, a partial example image is below:

Httpreports.png


One thing to look for is a log that is a lot larger than the others, as it might indicate excessive internet requests. Of course, they might all be normal, expected internet connections, but it takes someone familiar with the code and function to know.