Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "OSEE/ReqAndDesign"

(Requirements)
(Logging)
(45 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Logging ==
+
== Activity Logging and Monitoring ==
 
=== Requirements ===
 
=== Requirements ===
* shall handle thousands of log entries per second
+
* shall handle creation/update of fine-grained log entries for at least 500 concurrent users
* log entries shall be quickly accessible based on any combination of server, user, timestamp, log type, duration, status
+
* shall support logging by OSEE and other applications
 +
* the web of log entries related to an individual instance of a user request shall be able to be hierarcically related
 +
* log entries shall be quickly accessible based on any combination of source, user, timestamp, log type, duration, status
 
* log entries shall be accessible (especially) when an application server is unresponsive
 
* log entries shall be accessible (especially) when an application server is unresponsive
 
* log entries shall be available until they are deleted by an admin or admin policy (applied by server automatically)
 
* log entries shall be available until they are deleted by an admin or admin policy (applied by server automatically)
* at run-time logging shall be enabled/disabled based on any combination of user, source, and type
+
* at run-time logging shall be enabled/disabled based on any combination of user, source, log level, and type
 +
* access control shall be applied at the log entry type basis
  
 
=== Design ===
 
=== Design ===
id, timestamp, user, source, type_id, duration, status, details (maybe in JSON format)
+
* osee_activity db table
 +
:* Log entry in Java: long entryId, long parentId, long typeId, long startTime, long duration, long agentId, long status, String msgArgs
 +
:* entry_id - random long returned for log method call
 +
:* parent_id - id of entry used for grouping this and related entries. For root entries, it is the negative of session id of the client or the server id.  Ranges are used to group by client/server kind (IDE client, app server, rest client).
 +
:* type_id - foreign key to type_id in osee_log_type table
 +
:* start_time - long with ms since epoch
 +
:* duration - starts at -1 and is never updated if duration does not apply, otherwise updates when the associated job ends with duration in ms
 +
:* account_id - long account id (the account_id returned from account management services
 +
:* status:
 +
  0    initial value
 +
  1-99  percent complete
 +
  100  completed normally
 +
  101  completed abnormally
 +
:* msg_args newline separated list of strings used with String.format(msg_format, msg_args);
 +
* Each new log entry's parent_id, agent_id is mapped to the thread that created it (only the most recent mapping per thread is maintained)
 +
When an exception is thrown, it is logged as a child of the parent corresponding to the current thread.  If no mapping is found 
 +
ConcurrentHashMap<Thread, Pair<Long, Long>>()
  
log entry types are defined as tokens with a long and name (which is not in db)
+
* Log entry type in DB: type_id, log_level, software_unit, message_format
http://www.precisejava.com/javaperf/j2ee/JDBC.htm#JDBC104
+
** type_id - a fine-grained application defined type, random id, defined as tokens and stored in the db for cross application support
-7 is used for duration on instantaneous events
+
** log_level - as defined by java.util.logging.Level
otherwise the actual duration in ms is updated upon completion (in the meantime -1 is used)
+
** module - application defined name of the software unit that uses this log entry type
 +
** msg_format - format defined by [http://docs.oracle.com/javase/6/docs/api/java/util/Formatter.html java.util.Formatter] or if blank the raw message details are used directly
  
== Exception Handeling ==
+
* high performance
 +
** 2 ConcurrentHashMap are allocated with an initial configurable size: newLogEntires, updatedEntries
 +
** newly created log entries are added to newLogEntires using the entry_id as the key and the array of sql insert parameters as the value
 +
** updated log entries are checked for in newLogEntires and updated if they exist, otherwise the update map is checked and updated if exists, else added to updatedEntries
 +
** A timer tasks runs at a configurable (short) periodic rate and batch inserts the log entries in the insert map and then runs the updates. This means that any update to a log entry that occurs in less than this configured time will not require a database update (i.e. writing the duration of a short operation).  This also means only one thread writes to the log table per JVM.
 +
** new DrainingIterator(newLogEntires.values().iterator()) is used to iterate through the values and remove them one at a time during the batch insert
 +
** upon server shutdown must flush log
 +
** IDE client will directly use the same service that is used on the server
 +
** [http://stackoverflow.com/questions/8203864/the-best-concurrency-list-in-java data structure options]
 +
** [http://www.precisejava.com/javaperf/j2ee/JDBC.htm Optimize JDBC Performance]
 +
 
 +
 
 +
<source lang="java">
 +
Long createThreadEntry(long userId, Long typeId);
 +
 
 +
Long createThreadEntry(long userId, Long typeId, long parentId);
 +
 
 +
Long createEntry(Long typeId, Object... messageArgs);
 +
 
 +
Long createEntry(Long typeId, Long parentId, Object... messageArgs);
 +
 
 +
void updateEntry(Long entryId, Long status);
 +
 
 +
Long createExceptionEntry(Throwable throwable);
 +
 
 +
</source>
 +
 
 +
* The first interface to the logging data can be the basic REST navigation
 +
 
 +
== Exception Handling ==
 
=== Requirements ===
 
=== Requirements ===
 +
* avoid unnecessary wrapping of exceptions
 
=== Design ===
 
=== Design ===
 +
[http://misko.hevery.com/2009/09/16/checked-exceptions-i-love-you-but-you-have-to-go/ Checked exceptions I love you, but you have to go]
 +
[http://jyops.blogspot.com/2012/03/why-should-you-use-unchecked-exceptions.html Why should you use Unchecked exceptions over Checked exceptions]
 +
[http://convales.blogspot.com/2012/09/clean-code-by-example-checked-versus.html Clean Code by Example: Checked versus unchecked exceptions]
 +
 +
* Use application specific exceptions that extend RuntimeException - application specific allows for setting exception breakpoints in the debugger
 +
* Do not declare any run-time exceptions in any method signatures

Revision as of 12:34, 19 April 2014

Activity Logging and Monitoring

Requirements

  • shall handle creation/update of fine-grained log entries for at least 500 concurrent users
  • shall support logging by OSEE and other applications
  • the web of log entries related to an individual instance of a user request shall be able to be hierarcically related
  • log entries shall be quickly accessible based on any combination of source, user, timestamp, log type, duration, status
  • log entries shall be accessible (especially) when an application server is unresponsive
  • log entries shall be available until they are deleted by an admin or admin policy (applied by server automatically)
  • at run-time logging shall be enabled/disabled based on any combination of user, source, log level, and type
  • access control shall be applied at the log entry type basis

Design

  • osee_activity db table
  • Log entry in Java: long entryId, long parentId, long typeId, long startTime, long duration, long agentId, long status, String msgArgs
  • entry_id - random long returned for log method call
  • parent_id - id of entry used for grouping this and related entries. For root entries, it is the negative of session id of the client or the server id. Ranges are used to group by client/server kind (IDE client, app server, rest client).
  • type_id - foreign key to type_id in osee_log_type table
  • start_time - long with ms since epoch
  • duration - starts at -1 and is never updated if duration does not apply, otherwise updates when the associated job ends with duration in ms
  • account_id - long account id (the account_id returned from account management services
  • status:
 0     initial value
 1-99  percent complete
 100   completed normally
 101   completed abnormally
  • msg_args newline separated list of strings used with String.format(msg_format, msg_args);
  • Each new log entry's parent_id, agent_id is mapped to the thread that created it (only the most recent mapping per thread is maintained)

When an exception is thrown, it is logged as a child of the parent corresponding to the current thread. If no mapping is found ConcurrentHashMap<Thread, Pair<Long, Long>>()

  • Log entry type in DB: type_id, log_level, software_unit, message_format
    • type_id - a fine-grained application defined type, random id, defined as tokens and stored in the db for cross application support
    • log_level - as defined by java.util.logging.Level
    • module - application defined name of the software unit that uses this log entry type
    • msg_format - format defined by java.util.Formatter or if blank the raw message details are used directly
  • high performance
    • 2 ConcurrentHashMap are allocated with an initial configurable size: newLogEntires, updatedEntries
    • newly created log entries are added to newLogEntires using the entry_id as the key and the array of sql insert parameters as the value
    • updated log entries are checked for in newLogEntires and updated if they exist, otherwise the update map is checked and updated if exists, else added to updatedEntries
    • A timer tasks runs at a configurable (short) periodic rate and batch inserts the log entries in the insert map and then runs the updates. This means that any update to a log entry that occurs in less than this configured time will not require a database update (i.e. writing the duration of a short operation). This also means only one thread writes to the log table per JVM.
    • new DrainingIterator(newLogEntires.values().iterator()) is used to iterate through the values and remove them one at a time during the batch insert
    • upon server shutdown must flush log
    • IDE client will directly use the same service that is used on the server
    • data structure options
    • Optimize JDBC Performance


 Long createThreadEntry(long userId, Long typeId);
 
 Long createThreadEntry(long userId, Long typeId, long parentId);
 
 Long createEntry(Long typeId, Object... messageArgs);
 
 Long createEntry(Long typeId, Long parentId, Object... messageArgs);
 
 void updateEntry(Long entryId, Long status);
 
 Long createExceptionEntry(Throwable throwable);
  • The first interface to the logging data can be the basic REST navigation

Exception Handling

Requirements

  • avoid unnecessary wrapping of exceptions

Design

Checked exceptions I love you, but you have to go Why should you use Unchecked exceptions over Checked exceptions Clean Code by Example: Checked versus unchecked exceptions

  • Use application specific exceptions that extend RuntimeException - application specific allows for setting exception breakpoints in the debugger
  • Do not declare any run-time exceptions in any method signatures

Back to the top