diff options
Diffstat (limited to 'doxygen/dox/ViewToolsJPSS.dox')
| -rw-r--r-- | doxygen/dox/ViewToolsJPSS.dox | 763 |
1 files changed, 763 insertions, 0 deletions
diff --git a/doxygen/dox/ViewToolsJPSS.dox b/doxygen/dox/ViewToolsJPSS.dox new file mode 100644 index 0000000..9c15395 --- /dev/null +++ b/doxygen/dox/ViewToolsJPSS.dox @@ -0,0 +1,763 @@ +/** @page ViewToolsJPSS Use Case: Examining a JPSS NPP File With HDF5 Tools +Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand +<hr> + +\section secViewToolsJPSSTOC Contents +<ul> +<li>\ref secViewToolsJPSSDeter</li> +<li>\ref secViewToolsJPSSView</li> +<li>\ref secViewToolsJPSSExam</li> +</ul> + +This tutorial illustrates how to use the HDF5 tools to examine NPP files from the JPSS project. The following files are discussed: +\code +SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 (<a href="https://support.hdfgroup.org/ftp/HDF5/examples/files/tutorial/SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5.gz">gzipped file</a>) +SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 (<a href="https://support.hdfgroup.org/ftp/HDF5/examples/files/tutorial/SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5.gz">gzipped file</a>) +\endcode + +\section secViewToolsJPSSDeter Determining File Contents +The first thing you may want to do is determine what is in your file. You can use the command-line tools or HDFView to do this: +\li @ref subsecViewToolsJPSSDeter_h5dump +\li @ref subsecViewToolsJPSSDeter_h5ls +\li @ref subsecViewToolsJPSSDeter_HDFView + +JPSS NPP files all contain two root level groups: +<table> +<tr> +<th>Group</th><th>Description</th> +</tr> +<tr> +<td>/All_Data +</td> +<td>Contains the raw data and optional geo-location information. +</td> +</tr> +<tr> +<td>/Data_Products +</td> +<td>Contains a dataset ending in <code style="background-color:whitesmoke;">Aggr</code> with +references to objects in the <code style="background-color:whitesmoke;">/All_Data</code> group. +Contains granules (datasets with a name ending in <code style="background-color:whitesmoke;">Gran_#</code>) +with references to selected regions in datasets under <code style="background-color:whitesmoke;">/All_Data</code>. +</td> +</tr> +</table> + +\subsection subsecViewToolsJPSSDeter_h5dump h5dump +With <code style="background-color:whitesmoke;">h5dump</code> you can see a list of the objects +in the file using the <code style="background-color:whitesmoke;">-n</code> option: +\code +h5dump -n <file> +\endcode + +For example: +\code +$ h5dump -n SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 +HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" { +FILE_CONTENTS { + group / + group /All_Data + group /All_Data/VIIRS-M9-SDR_All + dataset /All_Data/VIIRS-M9-SDR_All/ModeGran + dataset /All_Data/VIIRS-M9-SDR_All/ModeScan + dataset /All_Data/VIIRS-M9-SDR_All/NumberOfBadChecksums + dataset /All_Data/VIIRS-M9-SDR_All/NumberOfDiscardedPkts + dataset /All_Data/VIIRS-M9-SDR_All/NumberOfMissingPkts + dataset /All_Data/VIIRS-M9-SDR_All/NumberOfScans + dataset /All_Data/VIIRS-M9-SDR_All/PadByte1 + dataset /All_Data/VIIRS-M9-SDR_All/QF1_VIIRSMBANDSDR + dataset /All_Data/VIIRS-M9-SDR_All/QF2_SCAN_SDR + dataset /All_Data/VIIRS-M9-SDR_All/QF3_SCAN_RDR + dataset /All_Data/VIIRS-M9-SDR_All/QF4_SCAN_SDR + dataset /All_Data/VIIRS-M9-SDR_All/QF5_GRAN_BADDETECTOR + dataset /All_Data/VIIRS-M9-SDR_All/Radiance + dataset /All_Data/VIIRS-M9-SDR_All/RadianceFactors + dataset /All_Data/VIIRS-M9-SDR_All/Reflectance + dataset /All_Data/VIIRS-M9-SDR_All/ReflectanceFactors + group /Data_Products + group /Data_Products/VIIRS-M9-SDR + dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Aggr + dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0 + dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_1 + dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_2 + dataset /Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_3 + } +} +\endcode + +In the output above you can see that there are four granules (ending in +<code style="background-color:whitesmoke;">Gran_#</code>) in the +<code style="background-color:whitesmoke;">/Data_Products/VIIRS-M9-SDR/</code> group. + +\subsection subsecViewToolsJPSSDeter_h5ls h5ls +With <code style="background-color:whitesmoke;">h5ls</code> you can see a list of the objects in the +file using the <code style="background-color:whitesmoke;">-lr</code> +options. The <code style="background-color:whitesmoke;">h5ls</code> utility also shows shape and size +(dataspace) information about datasets. +\code +h5ls -lr <file> +\endcode + +For example: +\code +$ h5ls -lr SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 +/ Group +/All_Data Group +/All_Data/VIIRS-M9-SDR_All Group +/All_Data/VIIRS-M9-SDR_All/ModeGran Dataset {4/Inf} +/All_Data/VIIRS-M9-SDR_All/ModeScan Dataset {192/Inf} +/All_Data/VIIRS-M9-SDR_All/NumberOfBadChecksums Dataset {192/Inf} +/All_Data/VIIRS-M9-SDR_All/NumberOfDiscardedPkts Dataset {192/Inf} +/All_Data/VIIRS-M9-SDR_All/NumberOfMissingPkts Dataset {192/Inf} +/All_Data/VIIRS-M9-SDR_All/NumberOfScans Dataset {4/Inf} +/All_Data/VIIRS-M9-SDR_All/PadByte1 Dataset {12/Inf} +/All_Data/VIIRS-M9-SDR_All/QF1_VIIRSMBANDSDR Dataset {3072/Inf, 3200/Inf} +/All_Data/VIIRS-M9-SDR_All/QF2_SCAN_SDR Dataset {192/Inf} +/All_Data/VIIRS-M9-SDR_All/QF3_SCAN_RDR Dataset {192/Inf} +/All_Data/VIIRS-M9-SDR_All/QF4_SCAN_SDR Dataset {3072/Inf} +/All_Data/VIIRS-M9-SDR_All/QF5_GRAN_BADDETECTOR Dataset {64/Inf} +/All_Data/VIIRS-M9-SDR_All/Radiance Dataset {3072/Inf, 3200/Inf} +/All_Data/VIIRS-M9-SDR_All/RadianceFactors Dataset {8/Inf} +/All_Data/VIIRS-M9-SDR_All/Reflectance Dataset {3072/Inf, 3200/Inf} +/All_Data/VIIRS-M9-SDR_All/ReflectanceFactors Dataset {8/Inf} +/Data_Products Group +/Data_Products/VIIRS-M9-SDR Group +/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Aggr Dataset {16/Inf} +/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0 Dataset {16/Inf} +/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_1 Dataset {16/Inf} +/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_2 Dataset {16/Inf} +/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_3 Dataset {16/Inf} +\endcode +Note that the <code style="background-color:whitesmoke;">Inf</code> indicates that those datasets are appendable or unlimited in size. + +\subsection subsecViewToolsJPSSDeter_HDFView HDFView +If you open the file in HDFView, it will display the file and the root level groups within +it in the TreeView on the left. An HDF5 file is a folder with a "5" in the middle, followed +by the file name. There are two folders (groups) within the JPSS file +(<code style="background-color:whitesmoke;">All_Data/</code> and <code style="background-color:whitesmoke;">Data Products/</code>), +which you can select to see their contents: +<table> +<tr> +<td> +\image html hdfview-tree.png +</td> +</tr> +</table> + +If you click twice with the left-mouse button on a folder or group in the TreeView, the contents +of the folder will be listed. If you click twice on an object such as a dataset, a window with +the object's values will be displayed. + +Underneath the <code style="background-color:whitesmoke;">VIIRS-M1-SDR</code> folder are what HDF5 +calls datasets. The scarlet letter <code style="background-color:whitesmoke;">"A"</code> attached +to the group and datasets under <code style="background-color:whitesmoke;">Data_Products/</code> +indicates that there are attributes associated with them. + +\section secViewToolsJPSSView Viewing the User Block +All JPSS files contain a user block in XML with information about the file. The user block is an +optional space allocated at the beginning of an HDF5 file that is not interpreted by the HDF5 +library. Its size is a multiple of 512. + +Since the user block in JPSS files is stored in ASCII and it is stored at the beginning of an +HDF5 file, you could use a text editor or viewer to examine it. However, there are HDF5 utilities +that can help with this: +<table> +<tr> +<th>Utility</th><th>Description</th> +</tr> +<tr> +<td>h5unjam +</td> +<td>Extracts a user block from an HDF5 file +</td> +</tr> +<tr> +<td>h5dump +</td> +<td>The -B (--superblock) option displays the size of the user block in an HDF5 file +</td> +</tr> +</table> + +\subsection subsecViewToolsJPSSView_h5unjam h5unjam +The \ref secViewToolsEditAdd tutorial topic discusses the use of the +<code style="background-color:whitesmoke;">h5jam</code> and <code style="background-color:whitesmoke;">h5unjam</code> +utilities for adding or removing a user block from a file. An input HDF5 file +(<code style="background-color:whitesmoke;">-i</code>), output HDF5 file +(<code style="background-color:whitesmoke;">-o</code>), and user block text file +(<code style="background-color:whitesmoke;">-u</code>) can be specified with these tools. You can use the +<code style="background-color:whitesmoke;">h5unjam</code> tool to extract and view the user block in a JPSS file: +\code +h5unjam -i <Input HDF5 File> -o <Output HDF5 File> -u <User Block File> +\endcode + +For example this command will extract the user block into the file UB.xml: +\code +$ h5unjam -i SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 + -o svm09-noUB.h5 -u UB.xml +\endcode + +The input HDF5 file remains unchanged. The output HDF5 file will not contain the user block. +The <code style="background-color:whitesmoke;">UB.xml</code> file contains the user block +which can be viewed with a browser. + +\subsection subsecViewToolsJPSSView_h5dump h5dump +The h5dump utility has the <code style="background-color:whitesmoke;">-B (--superblock)</code> option for displaying the superblock in an HDF5 file. +The superblock contains information about the file such as the file signature, file consistency flags, +the number of bytes to store addresses and size of an object, as well as the size of the user block: +\code +h5dump -B (--superblock) +\endcode + +Below is an example (Unix): +\code +$ h5dump -B -H SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 | more +HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" { +SUPER_BLOCK { + SUPERBLOCK_VERSION 0 + FREELIST_VERSION 0 + SYMBOLTABLE_VERSION 0 + OBJECTHEADER_VERSION 0 + OFFSET_SIZE 8 + LENGTH_SIZE 8 + BTREE_RANK 16 + BTREE_LEAF 4 + ISTORE_K 32 + USER_BLOCK { + USERBLOCK_SIZE 1024 + } +} +\endcode + +Once you have the size of the user block, you can extract it from the file using system commands. +For example, on Unix platforms you can use the head command-line tool: +\code +head -c <USERBLOCK_SIZE> <JPSS File> >& USERBLOCK.xml +\endcode + +There are Unix tools for Windows that may work, such as <a href="http://gnuwin32.sourceforge.net/packages/coreutils.htm">CoreUtils for Windows</a>. + +\section secViewToolsJPSSExam Examining a Granule +<ul> +<li>@ref subsecViewToolsJPSSExam_h5dump<br /> +<ul> +<li>@ref subsubsecViewToolsJPSSExam_h5dumpRegRef</li> +<li>@ref subsubsecViewToolsJPSSExam_h5dumpQuality</li> +<li>@ref subsubsecViewToolsJPSSExam_h5dumpProps</li> +</ul></li> +<li>@ref subsecViewToolsJPSSExamr_HDFView</li> +</ul> + +\subsection subsecViewToolsJPSSExam_h5dump h5dump +There are several options that you may first want to use when examining a granule with h5dump: +<table> +<tr> +<th>Option</th><th>Description</th> +</tr> +<tr> +<td>-H, --header +</td> +<td>Prints header (metadata) information only +</td> +</tr> +<tr> +<td>-d D, --dataset=D +</td> +<td>Specifies the granule dataset +</td> +</tr> +<tr> +<td>-A 0, --onlyattr=0 +</td> +<td>Suppresses attributes +</td> +</tr> +<tr> +<td>-p, --properties +</td> +<td>Show properties of datasets +(See Properties) +</td> +</tr> +</table> + +You would specify the dataset (<code style="background-color:whitesmoke;">-d D</code>) and the +<code style="background-color:whitesmoke;">-H</code> options to view the metadata associated with +a specific granule. There are many attributes associated with a granule and +<code style="background-color:whitesmoke;">-A 0</code> can be used to suppress those. + +For example: +\code +h5dump -H -A 0 -d "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0" + SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 +\endcode + +This command displays: +\code + HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" { + DATASET "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0" { + DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG } + DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) } + } + } +\endcode + +To see the actual contents of the granule remove the <code style="background-color:whitesmoke;">-H</code> option: +\code +h5dump -A 0 -d "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0" + SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 +\endcode + +The above command displays: +\code +HDF5 "SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5" { +DATASET "/Data_Products/VIIRS-M9-SDR/VIIRS-M9-SDR_Gran_0" { + DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG } + DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) } + DATA { + DATASET /All_Data/VIIRS-M9-SDR_All/Radiance {(0,0)-(767,3199)}, + DATASET /All_Data/VIIRS-M9-SDR_All/Reflectance {(0,0)-(767,3199)}, + DATASET /All_Data/VIIRS-M9-SDR_All/ModeScan {(0)-(47)}, + DATASET /All_Data/VIIRS-M9-SDR_All/ModeGran {(0)-(0)}, + DATASET /All_Data/VIIRS-M9-SDR_All/PadByte1 {(0)-(2)}, + DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfScans {(0)-(0)}, + DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfMissingPkts {(0)-(47)}, + DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfBadChecksums {(0)-(47)}, + DATASET /All_Data/VIIRS-M9-SDR_All/NumberOfDiscardedPkts {(0)-(47)}, + DATASET /All_Data/VIIRS-M9-SDR_All/QF1_VIIRSMBANDSDR {(0,0)-(767,3199)}, + DATASET /All_Data/VIIRS-M9-SDR_All/QF2_SCAN_SDR {(0)-(47)}, + DATASET /All_Data/VIIRS-M9-SDR_All/QF3_SCAN_RDR {(0)-(47)}, + DATASET /All_Data/VIIRS-M9-SDR_All/QF4_SCAN_SDR {(0)-(767)}, + DATASET /All_Data/VIIRS-M9-SDR_All/QF5_GRAN_BADDETECTOR {(0)-(15)}, + DATASET /All_Data/VIIRS-M9-SDR_All/RadianceFactors {(0)-(1)}, + DATASET /All_Data/VIIRS-M9-SDR_All/ReflectanceFactors {(0)-(1)} + } +} +} +\endcode + +As you can see in the output above, the datatype for this dataset is: +\code +DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG } +\endcode + +This indicates that it is a dataset specifically for storing references to regions (or subsets) +in other datasets. The dataset contains 16 such references, and more can be added to it, as +indicated by the dataspace (in other words it is unlimited): +\code +DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) } +\endcode + +\subsubsection subsubsecViewToolsJPSSExam_h5dumpRegRef Viewing a Region Reference +What if we wanted to look at the <code style="background-color:whitesmoke;">NumberOfScans</code> data for a specific granule in a file? + +First, we may be interested in determining whether the scans were done at night or in the day. If a scan was at night, there will be no data. + +The attribute <code style="background-color:whitesmoke;">N_Day_Night_Flag</code> is used to determine when the scan was done. If you don't know where this attribute is located, you can use the <code style="background-color:whitesmoke;">-N</code> option to search for it in the file. If you were to run this command on the <code style="background-color:whitesmoke;">SVM09</code> file used above, you would see that the <code style="background-color:whitesmoke;">N_Day_Night_Flag</code> attribute has a value of <code style="background-color:whitesmoke;">Night</code> for the four granules in the file. Indeed, if you actually examine the <code style="background-color:whitesmoke;">NumberOfScans</code> data, you will see that only fill values are written. + +For that reason we will examine the <code style="background-color:whitesmoke;">NumberOfScans</code> data for the <code style="background-color:whitesmoke;">SVMO1</code> file below, as it was obtained during the day: +\code +h5dump -N N_Day_Night_Flag SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +\endcode + +It displays: +\code +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +ATTRIBUTE "N_Day_Night_Flag" { + DATATYPE H5T_STRING { + STRSIZE 4; + STRPAD H5T_STR_NULLTERM; + CSET H5T_CSET_ASCII; + CTYPE H5T_C_S1; + } + DATASPACE SIMPLE { ( 1, 1 ) / ( 1, 1 ) } + DATA { + (0,0): "Day" + } +} +} +\endcode + +There is just one granule in this <code style="background-color:whitesmoke;">SVM01</code> file, as shown below: +\code +$ h5dump -n SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +FILE_CONTENTS { + group / + group /All_Data + group /All_Data/VIIRS-M1-SDR_All + dataset /All_Data/VIIRS-M1-SDR_All/ModeGran + dataset /All_Data/VIIRS-M1-SDR_All/ModeScan + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfBadChecksums + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfDiscardedPkts + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfMissingPkts + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfScans + dataset /All_Data/VIIRS-M1-SDR_All/PadByte1 + dataset /All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR + dataset /All_Data/VIIRS-M1-SDR_All/QF2_SCAN_SDR + dataset /All_Data/VIIRS-M1-SDR_All/QF3_SCAN_RDR + dataset /All_Data/VIIRS-M1-SDR_All/QF4_SCAN_SDR + dataset /All_Data/VIIRS-M1-SDR_All/QF5_GRAN_BADDETECTOR + dataset /All_Data/VIIRS-M1-SDR_All/Radiance + dataset /All_Data/VIIRS-M1-SDR_All/RadianceFactors + dataset /All_Data/VIIRS-M1-SDR_All/Reflectance + dataset /All_Data/VIIRS-M1-SDR_All/ReflectanceFactors + group /Data_Products + group /Data_Products/VIIRS-M1-SDR + dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Aggr + dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0 + } +} +\endcode + +Now examine the references in the <code style="background-color:whitesmoke;">VIIRS-M1-SDR_Gran_0</code> granule +\code +$ h5dump -A 0 -d "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" + SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +DATASET "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" { + DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG } + DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) } + DATA { + DATASET /All_Data/VIIRS-M1-SDR_All/Radiance {(0,0)-(767,3199)}, + DATASET /All_Data/VIIRS-M1-SDR_All/Reflectance {(0,0)-(767,3199)}, + DATASET /All_Data/VIIRS-M1-SDR_All/ModeScan {(0)-(47)}, + DATASET /All_Data/VIIRS-M1-SDR_All/ModeGran {(0)-(0)}, + DATASET /All_Data/VIIRS-M1-SDR_All/PadByte1 {(0)-(2)}, + DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfScans {(0)-(0)}, + DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfMissingPkts {(0)-(47)}, + DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfBadChecksums {(0)-(47)}, + DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfDiscardedPkts {(0)-(47)}, + DATASET /All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR {(0,0)-(767,3199)}, + DATASET /All_Data/VIIRS-M1-SDR_All/QF2_SCAN_SDR {(0)-(47)}, + DATASET /All_Data/VIIRS-M1-SDR_All/QF3_SCAN_RDR {(0)-(47)}, + DATASET /All_Data/VIIRS-M1-SDR_All/QF4_SCAN_SDR {(0)-(767)}, + DATASET /All_Data/VIIRS-M1-SDR_All/QF5_GRAN_BADDETECTOR {(0)-(15)}, + DATASET /All_Data/VIIRS-M1-SDR_All/RadianceFactors {(0)-(1)}, + DATASET /All_Data/VIIRS-M1-SDR_All/ReflectanceFactors {(0)-(1)} + } +} +} +\endcode + +In the output above, you can see that the <code style="background-color:whitesmoke;">NumberOfScans</code> +reference is the sixth reference in the granule counting from the top. + +The list of references shown above is a 0-based index to the dataset. Therefore, to specify +<code style="background-color:whitesmoke;">NumberOfScans</code>, enter a start offset of +<code style="background-color:whitesmoke;">5</code> for the <code style="background-color:whitesmoke;">-s</code> +option (the sixth reference minus 1). To see the region reference data, use the <code style="background-color:whitesmoke;">-R</code> option. + +This command will display the data in the <code style="background-color:whitesmoke;">NumberOfScans</code> region reference: +\code +h5dump -A 0 -d "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" -s 5 -R + SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +\endcode + +It displays the number of scans (48): +\code +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +DATASET "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" { + DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG } + DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) } + SUBSET { + START ( 5 ); + STRIDE ( 1 ); + COUNT ( 1 ); + BLOCK ( 1 ); + DATA { + (5): DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfScans { + (5): REGION_TYPE BLOCK (0)-(0) + (5): DATATYPE H5T_STD_I32BE + (5): DATASPACE SIMPLE { ( 1 ) / ( H5S_UNLIMITED ) } + (5): DATA { + (0): 48 + (5): } + (5): } + } + } +} +} +\endcode + +The <code style="background-color:whitesmoke;">-s</code> option may be familiar as one of the options +that was described in the \ref secViewToolsViewSub tutorial topic. The other subsetting options are not included, +indicating that the default values are used. + +If you leave off the <code style="background-color:whitesmoke;">-R</code> option, you will see the subset selection, but not the data: +\code +$ h5dump -A 0 -d "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" -s 5 + SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +DATASET "/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0" { + DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG } + DATASPACE SIMPLE { ( 16 ) / ( H5S_UNLIMITED ) } + SUBSET { + START ( 5 ); + STRIDE ( 1 ); + COUNT ( 1 ); + BLOCK ( 1 ); + DATA { + DATASET /All_Data/VIIRS-M1-SDR_All/NumberOfScans {(0)-(0)} + } + } +} +} +\endcode + +\subsubsection subsubsecViewToolsJPSSExam_h5dumpQuality Viewing a Quality Flag +The quality flags in an NPP file can be viewed with h5dump using the <code style="background-color:whitesmoke;">-M</code> +option. Quality flags are packed into each integer value in a quality flag dataset. Quality flag datasets in NPP +files begin with the letters <code style="background-color:whitesmoke;">QF</code>. + +In the following NPP file, there are five Quality Flag datasets +(<code style="background-color:whitesmoke;">/All_Data/VIIRS-M1-SDR_All/QF*</code>): +\code +$ h5dump -n SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +FILE_CONTENTS { + group / + group /All_Data + group /All_Data/VIIRS-M1-SDR_All + dataset /All_Data/VIIRS-M1-SDR_All/ModeGran + dataset /All_Data/VIIRS-M1-SDR_All/ModeScan + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfBadChecksums + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfDiscardedPkts + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfMissingPkts + dataset /All_Data/VIIRS-M1-SDR_All/NumberOfScans + dataset /All_Data/VIIRS-M1-SDR_All/PadByte1 + dataset /All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR + dataset /All_Data/VIIRS-M1-SDR_All/QF2_SCAN_SDR + dataset /All_Data/VIIRS-M1-SDR_All/QF3_SCAN_RDR + dataset /All_Data/VIIRS-M1-SDR_All/QF4_SCAN_SDR + dataset /All_Data/VIIRS-M1-SDR_All/QF5_GRAN_BADDETECTOR + dataset /All_Data/VIIRS-M1-SDR_All/Radiance + dataset /All_Data/VIIRS-M1-SDR_All/RadianceFactors + dataset /All_Data/VIIRS-M1-SDR_All/Reflectance + dataset /All_Data/VIIRS-M1-SDR_All/ReflectanceFactors + group /Data_Products + group /Data_Products/VIIRS-M1-SDR + dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Aggr + dataset /Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0 + } +} +\endcode + +The flags in this particular dataset happen to be stored in every two bits of each quality flag dataset +element, and the values range from 0 to 2. In other words, to see the quality flag values for this +dataset, these bits would be examined: 0 and 1, 2 and 3, 4 and 5, or 6 and 7 (This information was +obtained from the Product Profile XML File.) + +For example, bits 0 and 1 in the <code style="background-color:whitesmoke;">VQF1_VIIRSMBANDSDR</code> dataset specify the flag that +"Indicates calibration quality due to bad space view offsets, OBC view offsets, etc or use of a +previous calibration view". It has 3 values: Good (0), Poor (1), or No Calibration (2). + +The <code style="background-color:whitesmoke;">-M</code> option is used to specify the quality +flag bit offset (<code style="background-color:whitesmoke;">O</code>) and length (<code style="background-color:whitesmoke;">L</code>): +\code +h5dump -d DATASET -M O,L FILE +\endcode + +To view the first quality flag (0-1) in a 5 x 6 subset of the <code style="background-color:whitesmoke;">QF1_VIIRSMBANDSDR</code> dataset, specify: +\code +h5dump -d "/All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR[0,0;;5,6;]" + -M 0,2 SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +\endcode + +This outputs: +\code +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +DATASET "/All_Data/VIIRS-M1-SDR_All/QF1_VIIRSMBANDSDR" { + DATATYPE H5T_STD_U8BE + DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } + PACKED_BITS OFFSET=0 LENGTH=2 + SUBSET { + START ( 0, 0 ); + STRIDE ( 1, 1 ); + COUNT ( 5, 6 ); + BLOCK ( 1, 1 ); + DATA { + (0,0): 2, 2, 2, 2, 2, 2, + (1,0): 2, 2, 2, 2, 2, 2, + (2,0): 0, 0, 0, 0, 0, 0, + (3,0): 0, 0, 0, 0, 0, 0, + (4,0): 0, 0, 0, 0, 0, 0 + } + } +} +} +\endcode + +To view more than one quality flag at a time simply add the bit offset and length values to +<code style="background-color:whitesmoke;">-M</code>, separated by commas. For example, this +<code style="background-color:whitesmoke;">-M</code> option specifies bits 0-1 and 2-3: +\code +h5dump -d DATASET -M 0,2,2,2 FILE +\endcode + +\subsubsection subsubsecViewToolsJPSSExam_h5dumpProps Properties +To view properties of a specific dataset with <code style="background-color:whitesmoke;">h5dump</code> +use the <code style="background-color:whitesmoke;">-p</code> option along with the +<code style="background-color:whitesmoke;">-d</code> option. Depending on the number of attributes +and the amount of data, the <code style="background-color:whitesmoke;">-A 0</code> and +<code style="background-color:whitesmoke;">-H</code> options can also be specified to suppress +printing of attributes and data values: +\code +h5dump -p -H -A 0 -d DATASET +\endcode + +The <code style="background-color:whitesmoke;">-p</code> option shows any compression filters +associated with a dataset, as well as layout and fill value information. This option can be helpful +in diagnosing performance and other issues. + +As an example, examine the <code style="background-color:whitesmoke;">/All_Data/VIIRS-M1-SDR_All/Radiance</code> +dataset in the <code style="background-color:whitesmoke;">SVM01</code> file: +\code +$ h5dump -p -H -A 0 -d "/All_Data/VIIRS-M1-SDR_All/Radiance" + SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +HDF5 "SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5" { +DATASET "/All_Data/VIIRS-M1-SDR_All/Radiance" { + DATATYPE H5T_STD_U16BE + DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } + STORAGE_LAYOUT { + CHUNKED ( 768, 3200 ) + SIZE 4915200 + } + FILTERS { + NONE + } + FILLVALUE { + FILL_TIME H5D_FILL_TIME_IFSET + VALUE 65529 + } + ALLOCATION_TIME { + H5D_ALLOC_TIME_INCR + } +} +} +\endcode + +We can see that the chunk size for this dataset is 768 x 3200, and the storage size is 4915200. + +What if the chunk size were smaller? + +The dataset was modified to have a chunk size of 1 x 10, using the +<code style="background-color:whitesmoke;">h5repack</code> utility, as shown below. +\code +$ h5repack -l /All_Data/VIIRS-M1-SDR_All/Radiance:CHUNK=1x10 + SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 SVM01repack.h5 + +$ h5dump -p -H -A 0 -d "/All_Data/VIIRS-M1-SDR_All/Radiance" SVM01repack.h5 +HDF5 "SVM01repack.h5" { +DATASET "/All_Data/VIIRS-M1-SDR_All/Radiance" { + DATATYPE H5T_STD_U16BE + DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } + STORAGE_LAYOUT { + CHUNKED ( 1, 10 ) + SIZE 4915200 + } + FILTERS { + NONE + } + FILLVALUE { + FILL_TIME H5D_FILL_TIME_IFSET + VALUE 65529 + } + ALLOCATION_TIME { + H5D_ALLOC_TIME_INCR + } +} +} +\endcode + +In this case, the storage size of the dataset is the same, but the size of the file almost doubled!: +\code +$ ls -1sh +total 35M +12M SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 +23M SVM01repack.h5 +\endcode + +In general, the smaller the chunk size, the more chunks that HDF5 has to keep track of, which increases +the size of the file and can affect performance. + +\subsection subsecViewToolsJPSSExamr_HDFView HDFView +As mentioned previously, the structure of an HDF5 file is displayed in the TreeView on the left side of the HDFView screen, +and you can click on objects and have metadata information displayed on the right side. + +To discover more about the granule <code style="background-color:whitesmoke;">/Data_Products/VIIRS-M1-SDR/VIIRS-M1-SDR_Gran_0</code> +in the <code style="background-color:whitesmoke;">SVM01</code> file shown below in the TreeView, position +the mouse over the granule and click to select. Properties for the object is displayed on the right side of the HDFView screen. +You can see Datatype and Dataspace information on the <code style="background-color:whitesmoke;">General Object Info</code> +tab, any Attributes associated with the granulewill be on the +<code style="background-color:whitesmoke;">Object Attribute Info</code> tab. In the +<code style="background-color:whitesmoke;">General Object Info</code>, you can see that the dataset is a +Region Reference dataset, and that there are sixteen Region References in this dataset: +<table> +<tr> +<td> +\image html hdfview-prop.png +</td> +</tr> +</table> + +To examine the data in the granule, click twice on it with the left mouse button in the TreeView, +and it will open in a new window.: +<table> +<tr> +<td> +\image html hdfview-regref.png +</td> +</tr> +</table> + +If you click twice with the left mouse button on the fifth Region Reference +<code style="background-color:whitesmoke;">/All_Data/VIIRS-M1-SDR_All/NumberOfScans</code> a window +will pop up with the value(s) of the reference: +<table> +<tr> +<td> +\image html hdfview-regref2.png +</td> +</tr> +</table> + +You can also set a user option to automatically show the value(s) in a Region Reference. Under the +<code style="background-color:whitesmoke;">Tools</code> pull-down menu, select +<code style="background-color:whitesmoke;">User Options</code> and then select +<code style="background-color:whitesmoke;">HDF Settings</code> and then select +<code style="background-color:whitesmoke;">Show RegRef Values</code> in the +<code style="background-color:whitesmoke;">Data</code> section (see the middle of the image below): +<table> +<tr> +<td> +\image html hdfview-regrefval.png +</td> +</tr> +</table> + +Then you will automatically see the values of the Region Reference when you open it and select an entry: +<table> +<tr> +<td> +\image html hdfview-regref1.png +</td> +</tr> +</table> + +You can view and set quality flags by clicking the right mouse button over a quality flags dataset under +<code style="background-color:whitesmoke;">All_Data</code> and selecting +<code style="background-color:whitesmoke;">Open As</code> from the pop-up menu. In the middle of +the window that pops up, you will see where you can specify <code style="background-color:whitesmoke;">Bitmask</code> options. +<table> +<tr> +<td> +\image html hdfview-qf.png +</td> +</tr> +</table> + +<hr> +Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + +*/ |