|
|
| (73 intermediate revisions by 17 users not shown) |
| Line 1: |
Line 1: |
| − | == NetApp Unified Storage Appliance Report - Design Document ==
| + | iot is cool |
| | | | |
| | | | |
| − | Create a new Unified Storage Appliance Report - Initial Requirements
| + | {{cq|111}} |
| − | NetApp devices represent themselves in three different ways (depending on customer usage).
| + | |
| − | Combined, this is referred to as a Unified Storage Appliance. The three representations include:
| + | |
| − | - Traditional NAS filer (appliance storage is presented to clients as mounted file shares)
| + | |
| − | - FC block storage array (appliance provides FC attached blocks/volumes and looks just like a storage subsystem from this view)
| + | |
| − | - iSCSI block storage array (appliance provides iSCSI attached target volumes)
| + | |
| | | | |
| − | However, Aperi's data server looks at NetApp appliances only as NAS filers, ignoring its other two personalities.
| + | {{cq|111|2}} |
| − | To support a Unified Storage Appliance report, Aperi must gather information for all three representations.
| + | |
| | | | |
| − | The Aperi data component gathers information on NetApp from 2 sources
| |
| − | - Aperi data agent which sees the mounted file systems or Shares via NFS or CIFS views from a client perspective
| |
| − | - NetApp SNMP MIB from which we get some basic, high level asset and simple capacity information on the whole NetApp box.
| |
| − | There is a chance that this technique may produce inaccurate results.
| |
| | | | |
| − | The Unified Storage View needs to show the following type of capacity information (exact information and metrics will be determined DIRECTLY from what NetApp provides in their new vendor specific profile for this):
| + | __FORCETOC__ |
| − | 1. Show Appliance capacities - raw, used, reserved storage
| + | |
| − | 2. Label logical volumes as NAS, FC, iSCSI, or unassigned
| + | |
| − | 3. Total storage capacity used for NAS, iSCSI, FCP Blocks, snapshot, snapshot reserve, free, and unallocated
| + | |
| − | 4. Asset Information will already be available through information gathered in NAS Self-Contained Profile
| + | |
| − | 5. Configuration information on NetApp such as RAID and Cache configurations
| + | |
| | | | |
| − | Note that all of the above information will be available directly through SMI-S extension(s) from NetApp.
| |
| − | We will use our existing SMI-S support structure as a base and gather information from the new NetApp vendor specific profile(s).
| |
| − | The intent is to create a new Unified Storage Appliance Report that simply shows any and all information that NetApp gives to us in the vendor specific profile.
| |
| − | We will also request from NetApp, field titles and descriptions that we will directly map to the report.
| |
| − | So, basically, we take whatever NetApp hands us in the new SMIS profile and slap it into a report. The new vendor specific profile is being created specifically for this report.
| |
| | | | |
| − | We may wish to augment the information provided in the new profile with data we gather from other profiles such as the Block Services Profile, the SMI-S Self-Contained NAS profile, etc.
| |
| | | | |
| − | Narrowing the Scope of the Requirements to What Is Achievable
| + | = Testing IoT Commons = |
| − | Below are listed the items which, even though mentioned in the requirements, will not be put in this report:
| + | |
| − | - "Capacity - reserved storage", "Cache configuration" - These are not clearly defined in the requirements and it is difficult to give meaning to them only from their names
| + | |
| − | - "Snapshot", "Snapshot Reserve" - NetApp SMI-S agent does not offer information about snapshots.
| + | |
| − | - "Total storage capacity used for NAS" - NetApp SMI-S agent does not fill in the FileSize property of the CIM_LogicalFile so the sizes of the exported directories and files (i.e. storage capacity used for NAS) cannot be computed.
| + | |
| − | Everything else will be included in this report.
| + | |
| − | This document is created based on NetApp SMI-S Agent nasmis1.1X27 (the latest version as I write this)
| + | |
| | | | |
| − | SMI-S Agent Data Collection
| + | {| border="1" cellpadding="5" cellspacing="0" |
| − | Most of the data needed by this report is collected by the implementation of "Array" and "NAS Self Contained" profiles.
| + | |'''Name''' |
| | + | |BLE in Java |
| | + | |- |
| | + | |'''Origin''' |
| | + | |Kura |
| | + | |- |
| | + | |'''Description''' |
| | + | |Lorem ipsum Consectetur velit cupidatat proident cupidatat esse cillum dolore minim non dolore elit nulla elit cillum Ut eiusmod adipisicing tempor nostrud nisi id labore qui ullamco cupidatat ut minim laboris eu sunt culpa laboris. |
| | + | |- |
| | + | |'''Location''' |
| | + | |Source code: xxx<br>Binaries: yyy |
| | + | |- |
| | + | |'''Dependencies''' |
| | + | |X<br>Y<br>Z |
| | + | |- |
| | + | |'''Maturity''' |
| | + | |Mature Release in Kura 1.1 |
| | + | |} |
| | | | |
| − | The LUNs masking and mapping information (i.e which iSCSI initiator nodes or FC initiator ports have access to LUNs) is collected as part of LUN Masking and Mapping subprofile.
| |
| − | In the case of the iSCSI protocol the pathes between LUNs and initiator nodes are not saved in T_RES_DATA_PATH. This problem seems to get solved if "collectSAPAvailableForElement" and "collectDeviceSAPImplementation" steps are moved before "collectProtocollControllerForUnit" in org.eclipse.aperi.disk.collection.ProbeGenericMaskingMappingProcess. Side effects must be analyzed
| |
| − | The main classes which implement LUNs Masking and Mapping protocol are:
| |
| − | - org.eclipse.aperi.disk.collection.ProbeGenericMaskingMappingProcess - defines and executes the steps necesserary for collecting the instances in LUN Masking and Mapping profile
| |
| − | - org.eclipse.aperi.disk.collection.MappingMaskingProcessor - collects data from the steps exectuted be ProbeGenericMaskingMappingProcess and saves it in the repository by calling appropriate mappers
| |
| − | The collected data is put in T_RES_CLIENT_SETTING, T_RES_MASKING_INFO, T_RES_PORT and T_RES_DATA_PATH.
| |
| − | The iSCSI iNodes are stored in T_RES_PORT. Side efects must be analyzed.
| |
| − | T_RES_Port.FormatName is always 1. Should be changed to reflect the different types of names stored in this table ( FC WWN, iSCSI node names)
| |
| | | | |
| | + | {{M2M_Scenario |
| | + | | identifier = 0001 |
| | + | | name = Dummy scenario |
| | + | | description = Blah blah |
| | + | | rel_scenario1 = 0002 |
| | + | | ref_problem1 = 0001 |
| | + | | ref_problem2 = 0008 |
| | + | }} |
| | | | |
| − | Corellation between Array Profile's Entities (Storage Extents, Pools, LUNs) and Self Contained NAS' Entities( Filesystems )
| + | {{M2M_Scenario |
| − | The [[aggregate]] a volume resides on will be found using storage extents, skipping over the logical disk.
| + | | identifier = 0002 |
| | + | | name = Dummy scenario2 |
| | + | | description = Blah blah |
| | + | }} |
| | | | |
| − | A full path from a Filesystem (i.e volume which can be traditional or FlexVol) to its containing storage pool ( aggregate ) is :
| + | {{M2M_Scenario |
| − | CIM_FileSystem (ONTAP_LocalFS) ---------------- CIM_ResidesOnExtent(ONTAP_LocalFSResidesOnExtent) --------------------------------------------CIM_LogicalDisk (ONTAP_LogicalDisk)
| + | | identifier = 0003 |
| − | CIM_LogicalDisk (ONTAP_LogicalDisk)------------ CIM_BasedOn (ONTAP_FlexVolBasedOnExtent|ONTAP_BasedOn) ---------------------------------------ONTAP_FlexVolExtent|ONTAP_ConcreteExtent
| + | | name = Dummy scenario3 |
| − | ONTAP_FlexVolExtent|ONTAP_ConcreteExtent ------ CIM_ConcreteComponent (ONTAP_ConcreteFlexVolComponent|ONTAP_ConcreteComponent) -------------- CIM_StoragePool (ONTAP_ConcretePool)
| + | | description = Blah blah |
| − | Note1: '|' means 'or' and separates the classes used to represent FlexVols from the classes used to represent traditional volumes
| + | }} |
| | | | |
| − | The path between ONTAP_LocalFS and ONTAP_FlexVolExtent|ONTAP_ConcreteExtent can be shortcuted using the condition: ONTAP_LocalFS.Root == '/vol/'+ ONTAP_FlexVolExtent.Name, for FlexVols and ONTAP_LocalFS.Root == '/vol/'+ ONTAP_ConcreteExtent.Name for traditionale volumes.
| + | = blah = |
| − | ONTAP_FlexVolExtent|ONTAP_ConcreteExtent and their associations to ONTAP_Concrete Pools are collected by the Array Profile implementation and stored in T_RES_STORAGE_EXTENT.
| + | |
| − | The storage extents containing volumes are collected in the step named "collectExtentsFromPool", which is a substep of "collectPoolsFromComputerSystem" (see org.eclipse.aperi.disk.collection.ProbeNetAppSubsystemProcess). The association used is ONTAP_ConcretePool:CIM_ConcreteComponent:CIM_StorageExtent (starting from an ONTAP_ConcretePool and following the CIM_ConcreteComponent) return all the instances of type CIM_ConcreteComponent. ( see org.eclipse.aperi.disk.collection.ProbeGenericArraySubsystemProcess.process() ). The FlexVolExtents are stored in the repository using the mapper org.eclipse.aperi.infrastructure.mapping.SMISONTAP_FlexVolExtentToDBMapper. The ONTAP_ConcreteExtent are stored using the mapper org.eclipse.aperi.infrastructure.mapping.SMISONTAP_ConcreteExtentToDBMapper.
| + | |
| | | | |
| − | The LUNs and the volumes they are part of will be correlated using the LUNs' names, volumes mount point and the serial number of their storage subsystem. The LUN names are in fact absolute file names of which prefix is the mount point of the volume. For example the LUN named /vol/vol0/lun0 is part of the WAFL filesystem mounted at /vol/vol0.
| + | x |
| | | | |
| | + | = blah = |
| | | | |
| − | High Level Description of the Report
| + | x2 |
| − | I imagine the report as a series of drilldown reports structured on four levels.
| + | |
| | | | |
| − | 1.) The top level report displays general information about a NetApp storage subsystem.
| + | = blah = |
| − | A row in this report has the following columns:
| + | |
| − | - two drilldown icons for accessing "Volume Details" and "Aggregate Details" reports
| + | |
| − | - "Filer Name"
| + | |
| − | - "Capacity: Row Storage" - total disk capacities
| + | |
| − | - "Capacity: Assigned Storage" - sum of disk capacities already assigned to aggregates
| + | |
| − | - "Capacity: Free Space on Aggregates" - space that has not been consumed by the existing FlexVols and can be used to create new FlexVols. A traditional volume consumes all the space in its aggregate.
| + | |
| − | - "Capacity: Available Storage" - sum of hot spare disk capacities
| + | |
| − |
| + | |
| | | | |
| − | 2.) Second level reports
| + | z |
| − | 2.1) Volume Details. Each row corresponds to a volume and contains the following columns:
| + | |
| − | - one drilldown icon for accessing "Volume Content" report
| + | |
| − | - "Filer Name"
| + | |
| − | - "Volume Mount Point"
| + | |
| − | - "Volume Type" - flexible or traditional
| + | |
| − | - "Total Volume Space" -
| + | |
| − | - "Free Space" - free space on the volume
| + | |
| − | - "Space Occupied by LUNs" - total space assigned to LUNs
| + | |
| − | - "Number of LUNs" - number of LUNs created on this volume
| + | |
| − | - "Number of Share" - total number of files shares
| + | |
| − | - "Aggregate the Volume Resides On"
| + | |
| | | | |
| − | 2.2) Aggregate Details. The columns of this report are:
| |
| − | - "Filer Name"
| |
| − | - "Aggregate Name",
| |
| − | - "Aggregate Raid Level"
| |
| − | - "Aggregate Total Space"
| |
| − | - "Aggregate Free Space"
| |
| − | - "Number of disks in the Aggregate"
| |
| − |
| |
| | | | |
| − | 3.) Volume Content. Describes the file shares and LUNs that are part of the volume. The columns of this report are:
| + | {| class="wikitable" border="1" |
| − | - drilldown icon that will be activated only if the row represents a LUN that is mapped to hosts. This icon will drilldown to the "LUNs Mapping Details"
| + | |- |
| − | - "Filer Name"
| + | ! users |
| − | - "Volume Mount Point"
| + | |- |
| − | - "LUN / Share Name" - LUN name, which is an absolute file name or the export name ( the name with which the file is exported/shared )
| + | |
| − | - "Protocol" - CIFS or NFS for shares and iSCSI or FC for LUNs. A file can be shared using both CIFS and NFS and a LUN can be mapped both through FC and iSCSI.
| + | |
| − | - "Allocated Space" - for exported files and directories this will be "N/A" for LUNs will represent the LUN's capacity
| + | |
| | | | |
| − | 4.) LUNs Mapping Details. Displays what FC initiator ports or iSCSI initiator nodes have access to this LUN.
| + | |- |
| − | - "Filer Name"
| + | | id |
| − | - "Volume Mount Point"
| + | |- |
| − | - "LUN Name"
| + | |
| − | - "Protocol"
| + | |
| − | - "Host Node / Port name"
| + | |
| | | | |
| − | Implementation Details
| + | |- |
| | + | | user |
| | + | |- |
| | | | |
| − | Device Server
| + | |- |
| − | Steps "collectSAPAvailableForElement" and "collectDeviceSAPImplementation" will be moved before "collectProtocollControllerForUnit" in org.eclipse.aperi.disk.collection.ProbeGenericMaskingMappingProcess. Side effects must be analyzed
| + | | pwd |
| − | Analyze the side effects caused by storing the iSCSI node names in T_RES_PORT.
| + | |- |
| | | | |
| − | Gui and DataServer
| + | |} |
| − | The class org.eclipse.aperi.TStorm.server.guireq.GuiReportReq will be extended to support a new type of report, which will have five subtypes:
| + | |
| − | /** New Report type */
| + | |
| − | public static final int NETAPP_UNIFIED_STORAGE = 88
| + | |
| − | /** NETAPP_UNIFIED_STORAGE subreports */
| + | |
| − | public static final int NETAPP_UNIFIED_STORAGE_MAIN = 112;
| + | |
| − | public static final int NETAPP_UNIFIED_STORAGE_VOLUMES = 113;
| + | |
| − | public static final int NETAPP_UNIFIED_STORAGE_AGGR = 114;
| + | |
| − | public static final int NETAPP_UNIFIED_STORAGE_VOLCONTENT = 115;
| + | |
| − | public static final int NETAPP_UNIFIED_STORAGE_LUNS = 116;
| + | |
| | | | |
| − | Following classes will be added to support the new report:
| + | {{SoC_Idea |
| − | - org.eclipse.aperi.TStorm.gui.NetAppUnifiedStorageTable - GUI tables for displaying the reports
| + | | name = Create implementation for Foo |
| − | - org.eclipse.aperi.TStorm.common.NetAppUnifiedStorageAdjuster - is the data model behind NetAppUnifiedStorageTable
| + | | mentors = Erne, Bert |
| − | - org.eclipse.aperi.repository.report.RptNetAppUnifiedStorageReport - this class is instantiated in DataServer and executes the queries against the database to extract the data. In this class the queries for each subtype of the NetaApp Unified Storage report are defined and executed.
| + | | project = PDE |
| | + | | projectUrl = http://www.eclipse.org/pde |
| | + | | description = Fix the most popular bugs and feature requests: provide patches or fixes for any open bugzilla with at least 20 votes. |
| | + | | infoUrl = http://wiki.eclipse.org/Google_Summer_of_Code_2008 |
| | + | }} |
| | | | |
| − | New serialaizible objects will be defined only if needed to transfer data between Data Server and GUI.
| + | <source lang="xml"> |
| − | (I will add details here as I develop the code)
| + | <first> |
| | + | <second> |
| | + | <third> |
| | + | </third> |
| | + | </second> |
| | + | </first> |
| | + | </source> |
| | | | |
| | | | |
| − | Short Glossary of Terms
| + | <source lang="java"> |
| − | This document blends together NetApp, SMI-S and Aperi terms and I hope that this glossary will clarify their intended meaning .
| + | public class Foo { |
| | + | public Foo(String aParameter) { |
| | + | object.method(aParameter, 23); |
| | + | } |
| | + | } |
| | + | </source> |
| | | | |
| − | aggregate
| + | {{warning|Don't do that!}} |
| − | NetApp specific. It is the highest level of disks grouping. Lower levels are plexes and RAID groups, which are the basis for an aggregate.
| + | |
| − | A volume is created on an aggregate.
| + | |
| − | An aggregate is described by NetApp SMI-S agent using an instance of ONTAP_ConcretePool, which is derived from CIM_StoragePool.
| + | |
| − | An aggregate is represented as a storage pool in Aperi.
| + | |
| | | | |
| − | plex
| |
| − | NetApp specific. A plex is an assembly of RAID groups. NetApp defines mirroring at plex level. An aggregate has only one plex when it is unmirrored and two plexes when it is mirrored.
| |
| | | | |
| − | volume
| + | ;item 1 |
| − | NetApp specific. A volume is file system of type WAFL. The LUNs are created on a volume as files but exposed as block storage space. The file shares are also part of a volume.
| + | : definition 1 ([http://google.fr Google]) |
| − | Traditional Volume
| + | ;item 2 |
| − | NetApp specific. A traditional volume is a volume that occupies by itself an entire aggregate.
| + | : definition 2-1 ([http://google.fr Google]) |
| | + | : definition 2-2 ([http://google.fr Google]) |
| | | | |
| − | FlexVol Volume
| + | = CSS hacking = |
| − | NetApp specific. A FlexVol volume shares its containing aggregate with many other FlexVols.
| + | |
| | | | |
| − | LUN
| + | <css> |
| − | Logical unit of storage. Same thing with a Storage Volume
| + | #editme{ background-color: yellow; } |
| − | | + | #editme:before { content:'inserted: '; } |
| − | ONTAP
| + | #bodyContent{ position:relative; } |
| − | NetApp specific. It is the operating system which manages a NetApp storage system.
| + | #absbox { position:absolute; top: 0px; right: 0px; } |
| − | | + | </css> |
| − | hot spare disk
| + | <p id="editme">original text</p> |
| − | disks that have not been added yet to any aggregates and can be used to extend an aggregate, to create a new one or to replace a failed disk in an aggregate. ONTAP replaces automatically a failed disk with one of the hot spare disks.
| + | <div id="absbox">absolutely positioned content</div> |
| − | | + | some other text |
| − | filer
| + | |
| − | A shorthand for a NetApp storage system
| + | |
| − | | + | |
| − | file share
| + | |
| − | A directory or a file that is made available by a filer to clients through NFS or CIFS protocol
| + | |
| − | | + | |
| − | LUN masking
| + | |
| − | | + | |
| − | LUN mapping
| + | |
