Difference between revisions of "OSEE/ReqAndDesign"

Revision as of 15:39, 2 May 2016

Fast, versioned tuple service

Requirements

• Creation and deletion of tuples shall have transactional history
• tuple operations shall be near constant time
• shall efficiently support tuples of numbers and strings of length 2, 3, and 4

Uses Cases

• multiple tags per transaction with with history

 osee_tuple3(tx_key_type, branch_id, tx_id, attr_id, gamma_id)

• cross branch linking

 osee_tuple2(relation_type, a_art_id, b_art_id, gamma_id)

• user defined tables

 osee_tuple2(Long my_table_type, Long row_id, String row_data, gamma_id)

• the row_id allows the rows to be returned in order, also allows showing history of a given row
• the row_data stores the json representing the row data

Design

osee_tuple2 (tuple_type, e1, e2, gamma_id)
osee_tuple3 (tuple_type, e1, e2, e3, gamma_id)
osee_tuple4 (tuple_type, e1, e2, e3, e4, gamma_id)
primary key index: tuple_type, e1, e2[, e3, e4]
unique index: gamma_id

• gamma_id provides for branching and history
• elements can be of type number or string. A string is stored as a key to the osee_key_value table

Product Line Engineering

Requirements

• User can quickly select branch-view without requiring knowledge of corresponding version of product line. Select by branch-view name (i.e. Country X San) which is short hand for selecting My Product Line 2.0 branch with Country X view.
• View applicability has history and can vary independently by branch.
• External tools can join on view applicability in their sql queries
• need list of branch-view to show in branch selection dialogs

Design

• database design

• item applicability is fully transactional on all branches

osee_txs (branch_id, transaction_id, gamma_id, tx_current, mod_type, app_id)

Default value for app_id in osee_txs is 1 which means in all views (base). View artifacts are on the product line branches.

• view applicability

osee_tuple2 (view_app_tuple_type, view_id, app_id, app_text, gamma_id)

The view applicability tuples are on the product line branches thus maintaining full transactional history. The app_text is the applicability text (feature applicability text or the name of the of the view). Each view will be mapped to applicability id 1 (Base) so that base will be included for all configs by simply joining on the view_id.

• named branch-views

osee_tuple3 (branch-view_tuple_type, branch_id, view_id, branch-view_name, gamma_id)

Named branch-views are stored as tuples on the common branch.

• feature applicability definition

osee_tuple4 (app_feature_tuple_type, app_id, feature_id, value_attr_id, and_or_flag, gamma_id)

The Feature artifact has an attribute for each possible value. It also has a name, description, abbreviation, and type (single or multi-valued).

osee_tuple3 (compound_app_tuple_type, app_id, app_id, and_or_flag, gamma_id)

feature applicability involving more than one feature is stored as a list of feature applicability (recursive to any depth). -- return all items (refrenced by gamma_id) currently in a view of a branch select * from osee_txs txs, osee_view_app app where branch_id = ? and and tx_current = 1 and txs.app_id = app.app_id and view_id = ?;

applicability clause in SQL is only applied when using a branch view

 Long createThreadEntry(long userId, Long typeId);
 
 Long createThreadEntry(long userId, Long typeId, long parentId);
 
 Long createEntry(Long typeId, Object... messageArgs);
 
 Long createEntry(Long typeId, Long parentId, Object... messageArgs);
 
 void updateEntry(Long entryId, Long status);
 
 Long createExceptionEntry(Throwable throwable);

Activity Logging and Monitoring

Requirements

• shall handle creation/update of fine-grained log entries for at least 500 concurrent users
• shall support logging by OSEE and other applications
• the web of log entries related to an individual instance of a user request shall be able to be hierarcically related
• log entries shall be quickly accessible based on any combination of source, user, timestamp, log type, duration, status
• log entries shall be accessible (especially) when an application server is unresponsive
• log entries shall be available until they are deleted by an admin or admin policy (applied by server automatically)
• at run-time logging shall be enabled/disabled based on any combination of user, source, log level, and type
• access control shall be applied at the log entry type basis

Design

• osee_activity db table

• Log entry in Java: long entryId, long parentId, long typeId, long startTime, long duration, long agentId, long status, String msgArgs
• entry_id - random long returned for log method call
• parent_id - id of entry used for grouping this and related entries. For root entries, it is the negative of session id of the client or the server id. Ranges are used to group by client/server kind (IDE client, app server, rest client).
• type_id - foreign key to type_id in osee_log_type table
• start_time - long with ms since epoch
• duration - starts at -1 and is never updated if duration does not apply, otherwise updates when the associated job ends with duration in ms
• account_id - long account id (the account_id returned from account management services
• status:

 0     initial value
 1-99  percent complete
 100   completed normally
 101   completed abnormally

• msg_args newline separated list of strings used with String.format(msg_format, msg_args);

• Each new log entry's parent_id, agent_id is mapped to the thread that created it (only the most recent mapping per thread is maintained)

When an exception is thrown, it is logged as a child of the parent corresponding to the current thread. If no mapping is found ConcurrentHashMap<Thread, Pair<Long, Long>>()

• Log entry type in DB: type_id, log_level, software_unit, message_format
  • type_id - a fine-grained application defined type, random id, defined as tokens and stored in the db for cross application support
  • log_level - as defined by java.util.logging.Level
  • module - application defined name of the software unit that uses this log entry type
  • msg_format - format defined by java.util.Formatter or if blank the raw message details are used directly

• high performance
  • 2 ConcurrentHashMap are allocated with an initial configurable size: newLogEntires, updatedEntries
  • newly created log entries are added to newLogEntires using the entry_id as the key and the array of sql insert parameters as the value
  • updated log entries are checked for in newLogEntires and updated if they exist, otherwise the update map is checked and updated if exists, else added to updatedEntries
  • A timer tasks runs at a configurable (short) periodic rate and batch inserts the log entries in the insert map and then runs the updates. This means that any update to a log entry that occurs in less than this configured time will not require a database update (i.e. writing the duration of a short operation). This also means only one thread writes to the log table per JVM.
  • new DrainingIterator(newLogEntires.values().iterator()) is used to iterate through the values and remove them one at a time during the batch insert
  • upon server shutdown must flush log
  • IDE client will directly use the same service that is used on the server
  • data structure options
  • Optimize JDBC Performance

 Long createThreadEntry(long userId, Long typeId);
 
 Long createThreadEntry(long userId, Long typeId, long parentId);
 
 Long createEntry(Long typeId, Object... messageArgs);
 
 Long createEntry(Long typeId, Long parentId, Object... messageArgs);
 
 void updateEntry(Long entryId, Long status);
 
 Long createExceptionEntry(Throwable throwable);

• The first interface to the logging data can be the basic REST navigation

Exception Handling

Requirements

• avoid unnecessary wrapping of exceptions

Design

Checked exceptions I love you, but you have to go Why should you use Unchecked exceptions over Checked exceptions Clean Code by Example: Checked versus unchecked exceptions

• Use application specific exceptions that extend RuntimeException - application specific allows for setting exception breakpoints in the debugger
• Do not declare any run-time exceptions in any method signatures