\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{makecell}
\usepackage{multicol}

\title{WBD QC Tool Manual}
\author{Matthew Morriss, Patrick Longley, and Brittany Gold}
\date{February 2021}

\begin{document}

\maketitle
\tableofcontents


\section{Introduction}
These tools were developed by the personnel of the WBD team (Patrick Longley, Brittany Gold, and Matthew Morriss at the UT Water Science Center). They are largely written in ArcPy for use with ArcMap or ArcPro. As these two softwares use different versions of Python (2.x and 3.x respectively), we have tested these tools on both softwares.

All of these tools are packaged into a .pyt - python toolbox - which can be added to both ArcPro and ArcMap.  The tools are package into a toolbox called WBD automation tools.

\section{Requirements}
These tools were developed in Python 3 and require ArcGIS PRO.  All of the tools were tested with ArcGIS Pro 2.6.3 as of March 1, 2020.

TO RUN:
\begin{itemize}
    \item At least ArcPro GIS 2.6.X
    \item Python 3.X (comes with ArcPro)
\end{itemize}

\section{Download and Install}

 The most up to date software can be downoaded from the project GitLab page \url{https://code.usgs.gov/bgold/wbd-qc-tools}. As of this writing, this page is internal to the USGS and requires a USGS login to access.


\section{General Use}

\section{Manual vs. Tutorial}

This is a Manual, not a Tutorial. This is a document kept to be a reference for the various functions and tools developed to the aid the QC process of WBD products. We make limited suggestions as to a workflow for the tools, mostly with the use of a Flow Chart of the various functions (Figure \ref{fig:Figure1} and \ref{fig:Figure2}).

\section{Workflow}

Possible workflows through the functions provided  are outlined in Figures \ref{fig:Figure1} and \ref{fig:Figure2}.
In the sections that follow, descriptions of each function are provided and in some cases, possible outputs are included as figures using
the example datasets.

\begin{figure}
    \centering
    \includegraphics[scale = 0.55]{Figures/Figure_1_WBDFlowchart.pdf}
    \caption{Two example workflows: \textbf{1)} Download Data and \textbf{2)} Elevation Derived Hydrography tools}
    \label{fig:Figure1}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[scale = 0.7]{Figures/Figure_2_WBDFlowchart.png}
    \caption{Two example workflows: \textbf{1)} Geometry Checks and \textbf{2)} Attribution Check tools}
    \label{fig:Figure2}
\end{figure}

\section{Tools}

    \subsection{Attribution Tools}
        \begin{table}[h!]
            \begin{center}
            \caption{Summary of Attribution Tools.}
            \label{tab:table1}
            \begin{tabular}{|p{30mm}|l|l|l|} % <-- Alignments: 1st column left, 2nd middle and 3rd right, with vertical lines in between
                \hline
                \textbf{Tool name} & \textbf{Input datasets} &\textbf{Ouput datasets} & \textbf{New columns}\\
                \hline
                1) General Check
                    &\makecell{ WBD line fc \\ WBD polygon fc(s)}
                    & NA
                    & "\_flag" columns\\
                    \hline
                2) Line: TNMID/HUMod/LineSource Check
                    & \makecell{Old WBD line fc\\New WBD line fc\\Old WBD polygon fc\\New WBD line fc}
                    &NA
                    & \makecell{TNMID\_calc\\HUMod\_calc\\LineSource\_calc}\\
                \hline
                3) Line: HUDigit Check
                    & \makecell{New WBD polygon fc(s)\\Reference WBD polygon fc(s)\\New WBD Line fc}
                    & NA
                    & HUDigit\_calc\\
                \hline
                4) Polygon: ToHUC/HUType Check
                    & \makecell{New WBD polygon fc\\NHD flowline fc\\NHD area fc}
                    & \makecell{Outlets\\Frontal Outlets}
                    & ToHUC\_calc\\
                \hline
                5) Polygon: Name Check
                    & \makecell{New WBD polygon fc\\GNIS fc\\NHD waterbody fc\\NHD flowline fc}
                    & NA
                    & optional\\
                \hline
                6) Polygon: Name Update
                    & New WBD polygon fc
                    & NA
                    & NA\\
                \hline
                7) Polygon: ReferenceGNIS\_IDs Check
                    & \makecell{New WBD polygon fc\\GNIS fc}
                    & NA
                    & ReferenceGNIS\_IDs\_calc\\
                \hline
            \end{tabular}
            \end{center}
        \end{table}

        \subsubsection{General Check}
            \textit{Tool Description}
            
            This tool performs simple checks on all fields that are editted during the EDH process. 
            The following checks are performed:
            \begin{itemize}
                \item Checks that text fields match the correct regular expression
                    (are generally in the correct format).
                \item Checks that Huc, TNMID, and Name fields are unique.
                \item Checks that the Huc and ToHUC fields match the Huc and ToHUC fields in the parent feature class.
            \end{itemize}
            \textit{Inputs}
            \begin{itemize}
                \item Updated WBD line feature class.
                \item WBD polygon feature classes (more than one can be entered).
            \end{itemize}
            \textit{Outputs} None

            % TODO don' indent
            \textit{Output Columns}
                \begin{multicols}{2}
                    \begin{itemize}
                        \item Line: TNMID\_flag
                        \item Line: HUDigit\_flag
                        \item Line: LineSource\_flag
                        \item Line: HUMod\_flag
                        \item Polygon: Name\_flag
                        \item Polygon: TNMID\_flag
                        \item Polygon: ReferenceGNIS\_IDs\_flag
                        \item Polygon: States\_flag
                        \item Polygon: Huc\_flag
                        \item ToHUC\_flag
                        \item AreaSqKm\_flag
                        \item AreaAcres\_flag
                        \item NonContributingAreaAcres\_flag
                        \item NonContributingAreaSqKm\_flag
                    \end{itemize}
                \end{multicols}
            Each of these new "\_flag" columns will have one of three flags:
            \begin{itemize}
                \item \textbf{CORRECT:} the field is the correct format.
                \item \textbf{INCORRECT:} the field is not the correct format.
                \item \textbf{NODATA:} the field is blank or Null
            \end{itemize}

        \subsubsection{Polygon: TNMID/HUMod/LineSource Check}
            \textit{Tool Description}
            
            This tool compares the TNMID, HUMod, and LineSource fields from the original WBD Line featureclass
            to the updated WBD line featureclass. If the values in the new featureclass are
            different from the original, they are flagged as incorrect.
            \par
            Since WBD updates affect both the geometry and attribution of the WBD line feature class, it is difficult
            to determine exactly which lines to compare when testing if the updated attribution is different from the
            original attribution.  This tool overcomes this problem by performing a kind of "best guess" spatial join that
            minimizes the sum of the euclidean distance between line centroids. This determine which lines in the original
            featureclass should be compaired to the lines in the new featureclass.  Thic calculation is performed on a 
            per huc basis.  It should be noted that this algorythim is not perfect and will not work where:
            \begin{itemize}
                \item a huc code is retired,
                \item a new huc code is retired,
                \item the number of line segments that make up a polygon changes,
                \item along the outer boundary (if the boundary has not been subdivided into the correct number of line segments yet).
            \end{itemize}

            \textit{Inputs}
            \begin{itemize}
                \item Old WBD line featureclass
                \item Updated WBD line featureclass
                \item Old WBD polygon featureclass: (smallest possible HUC subdivisions.  Generally HUC12; unless HUC14s or HUC16s exist for the given area.)
                \item Updated WBD polygon featureclass: (smallest possible HUC subdivisions.  Generally HUC12; unless HUC14s or HUC16s exist for the given area.)
                \item Fields to check: Which fields to check.  By default LineSource is not checked since this value
                    is likely to change and still be correct.
                \item New subdivisions: This option should be selected when new subdivisions are created
                    (for example if there are newly created HUC14s or HUC16s).
                \item Fill missing values: Controls ifmissing values are filled (and flagged with "FILLED") or left empty
                    and flagged with "NODATA"
            \end{itemize}
            \textit{Outputs} None

            \textit{Output Columns}
                \begin{itemize}
                    \item TNMID\_calc: TNMID value transfered from the original WBD line featureclass (if selected 
                        in "Fields to check").
                    \item HUMod\_calc: HUMod valu transfered from the original WBD line featureclass (if selected 
                        in "Fields to check").
                    \item LineSource\_calc: LineSource value transfered from the original WBD line featureclass (if selected 
                        in "Fields to check").
                \end{itemize}
                This tool also updates the TNMID\_flag, HUMod\_flag, and LineSource\_flag fields generated by 
                the General Check tool.  If these fields are not present in the dataset they will be added by this tool.
                \par
                This tool uses the following flags in the "\_flag" columns:
                \begin{itemize}
                    \item \textbf{CORRECT:} the updated data matches the original data.
                    \item \textbf{FILLED:} the field was filled by the tool (only applicable if ``Fill missing values'' is selected).
                    \item \textbf{NOTCHECKED:} the field was not checked by this tool.
                    \item \textbf{INCORRECT:} the updated data does not match the original data.
                    \item \textbf{NODATA:} the field is blank or Null.
                \end{itemize}
                It should be noted that this tool compliments the general check tool and performs a more in depth check
                that builds upon the checks performed by the general check tool.  This tool will not overwrite preexisting
                WRONG or NODATA flags but can change CORRECT to WRONG or NOTCHECKED.  Additionaly if the user has added notes
                to one of the "\_flag" fields the tool will not overwrite their notes (or the flag).                

    
                \subsection{Download Data Tools}
    
    \subsection{Elevation Derived Hydrography Tools}
    
    \subsection{Geometry Checks}
    
        \subsubsection{Fill Depth and Polygon}
            \textit{Tool Description}
        
            This tool was written out of necessity to identify areas where the Arc \texttt{Fill} tool would create errors in the eventual \texttt{ArcHydro} processes. The primary case considered in this work was the presence of culverts, where filling an area upstream of a road would erroneously send the flowline around the culvert, when in actuality the flowline should go beneath the road. There are other instances wherein excessive filling could change the course of flowlines across a natural landscape (e.g. karst topography or very low relief wetland areas). 
            
            
            \textit{Inputs}
            \begin{itemize}
                \item Input Raster - Unfilled (projected with meters as linear unit)
                \item \texttt{Checkbox:} Providing Filled Raster? If providing an already filled raster (e.g. from NHDPlus HR)
                \item Filled Raster Location
            \end{itemize}
            
            \textit{Outputs}
            \begin{itemize}
                \item Filled Raster. A duplicate if a filled raster is provided.
                \item Fill-Depth Raster. The difference between original unfilled raster and the filled raster.
                \item Polygon. Polygon feature class of the areas that were filled in the original raster.
            \end{itemize}
            
            \begin{figure}
                \centering
                \includegraphics[scale = 0.7]{Figures/Fill_depth_example.jpg}
                \caption{Fill Depth Tool Example. \textbf{First Panel)} Open Street Map view of meanders on the Colorado River near Austin, Texas \textbf{Second Panel)} Elevations of the unfilled DEM. \textbf{Third Panel)} Fill Depth result from tool across study area. The areas highlighted in this figure will also became the polygons outlined in the polygon layer output.}
                \label{fig:my_label}
            \end{figure}
        
    
        \subsubsection{Low Relief Finder}
            \textit{Tool Description}
        
            This tool was developed out of a need while conducting plumbing reviews to identify areas of very low-relief where the Arc Hydro tools may not function well in their typically automated fashion. By finding areas that are significantly low-relief, this tool can guide the reviewers eyes to areas that could be problematic and need a closer inspection. This does not replace the need for the reviewer to be on the lookout for areas where Arc Hydro either crosses water-bodies or deviates from what the review determines is the drainage divide.
            
            This tool operates through a moving window, which calculates relief across a DEM. The \emph{default} behavior of the tool identifies areas that are 1-standard deviation below the mean relief in the landscape. This would equate to the areas with the lowest 15\% of relief in the landscape. The User can provide their own custom thresholds in units of standard deviation. The tools puts out a relief raster and a polygon feature class 
            
        
            \textit{Inputs}
            \begin{itemize}
                \item Input Raster (projected with meters as linear unit)
                \item Moving Window Radius (m)
                \item \texttt{Checkbox:} Providing Custom Threshold?
                \item Cutoff Threshold (standard deviations; e.g. 1, 2, 3) below the mean
                \item \texttt{Checkbox:} Run Second Iteration? Identify areas with even lower relief with a second pass of the tool.
            \end{itemize}
            
            \textit{Outputs}
            
            \begin{itemize}
                \item Output Raster - Raster and location 
                \item Output Polygon - Polygon feature class name and location for areas of low relief.
            \end{itemize}
            
            \begin{figure}
                \centering
                \includegraphics[scale = 0.7]{Figures/Low_relief_example.jpg}
                \caption{Low Relief Tool Example \textbf{First Panel)} Open Street Map view of example area in Central Texas, USA \textbf{Second Panel)} Elevations of DEM across example area \textbf{Third Panel)} Low Relief tool results. Areas identified as meeting tool qualifications highlighted in teal.}
                \label{fig:my_label}
            \end{figure}


    \subsection{Job Checkout}










\end{document}


