Carrie Browne wrote:
> 1) We need to know if the geospatial data clearinghouses
> are going to accept metadata created with ArcCatalog if it
> is producing errors in mp.  The geospatial clearinghouse
> that we contribute to uses mp as the criteria for posting.

It is my hope that clearinghouse nodes not rely exclusively
on mp's error messages as a measure of metadata quality.
Certain kinds of errors should be regarded as crucial, since
they inhibit further processing, but other errors are far
less serious.  I try to discourage people from becoming slaves
of mp because it's really the content of the metadata, not
the structure, that matters most.

That said, the structure can be a useful indicator to use
if you want to determine what information has been left out,
and the structure tells you where to find the answers to
specific questions that users will have.  So the error report
should not be disregarded.  In my version of the Top Ten
Metadata Mistakes, I prioritize error messages from most to
least serious like this: (1 = most, 8 = least serious)

  1. Indentation problems
  2. Unrecognized elements
  3. Misplaced elements
  4. Too many of some element
  5. Missing elements
  6. Empty elements
  7. Improper element values
  8. Warnings and upgrades

In general, any error of type 1-3 really must be fixed.
Errors of type 4 probably should be fixed, although I know
situations in which they should be tolerated.  In contrast,
errors of type 5-7 are far less serious, and while they
indicate that you're not providing people some information
that they might need, they don't represent situations in
which information will actually be lost as the metadata
gets moved from one program to another.  You can use err2html
to get a more readable presentation of the error report.

Real review of metadata should start with mp but should not
end there.  A person needs to read the metadata to determine
how well the information is written, especially whether some
important question is simply left unanswered.  One way to
carry out this review is to run mp but generate the FAQ-style
HTML output.  This procedure rewrites the metadata as a
series of plain-language questions, and uses the text of the
metadata to answer those questions.  Where an important
question is left unanswered, or where the answer doesn't seem
to fit the question or doesn't really answer the question,
some revision of the metadata should be required.

This process is the reverse of "Metadata in Plain Language",
my web pages designed to help people create metadata by
thinking of it as a set of questions.  That's on the web at
http://geology.usgs.gov/tools/metadata/tools/doc/ctc/

So the answer here is more complicated than some would like,
but touches more cleanly on the reality of the problem at
hand--what do we want users to know?  Standard metadata
emphasize that the same questions apply to many diverse kinds
of data that GIS and more generally, science, can deliver.

> 2) Assuming that mp will continue to be the standard, the
> mp program found the most errors in the Attribute and
> Entities section of the metadata body. Specifically within
> the definition, definition source, and domain values of
> attributes and entities that were created by the maker of
> the coverage.  Any time the Attribute or Entity is not ESRI
> appointed in the .aat and .pat tables ArcCatalog only pulls
> out the attribute name but no definition, definition
> source, or domain value which produces an error in mp. Do
> you have any suggestions on this?

What's happening here is that ArcCatalog is creating a
Detailed_Description, while fgdcmeta.aml is creating an
Overview_Description.  The FGDC standard allows either
of these or both of these.  Opinions differ on the value of
one or the other.  Since I'm not shy, I'll tell you I prefer
the Detailed_Description because it prompts you to explain
more about the attributes.  The Overview_Description is a
free-text field, and people have a tendency to put lots of
stuff there, but they also tend to leave out important things
like the units of the attributes, and the software doesn't
know that piece of information is missing.

That said, there are circumstances in which I favor using an
Overview_Description.  When the attributes are described fully
in a well-reviewed user's guide that is published on the web,
especially when the same data dictionary is used by lots of
different data sets (coverages), it makes sense to give a brief
outline of the information content and then use the
Entity_and_Attribute_Detail_Citation to give a URL that takes
the user to the master data dictionary.

I provide some tools that make it easier to populate a
Detailed_Description.  If you have a table that relates
your field values to their meanings (supposing the field
value is an abbreviation), you can use the Enumerated_Domain
helper at http://geology.usgs.gov/tools/metadata/tools/doc/ctc/edom.html
If you want to simply survey the contents of a DBF file,
you can use dbfmeta.  The results of either of these would
then be pasted (by you) into the metadata record using some
editing tool like Tkme.  DBFmeta has a web page at
http://geology.usgs.gov/tools/metadata/tools/doc/dbfmeta.html
Since it does the equivalent of ArcInfo's LISTUNIQUE, this
is also a good data-consistency check.  Note that DBFmeta
does some useful things, but like ArcCatalog, it only knows
what's in the DBF file, not what the attributes really mean,
or what the units are.

> Also, it was later
> learned that we can't figure out where these attributes are
> pulled from because every attribute table that we looked at
> for this coverage only had 1 of the 5 attributes in
> question listed. Do you know where ArcCatalog pulls this
> data from and how I can look at that data that it is
> pulling it from? I need to know what the data is before I
> can manually produce a definition, source, and domain.

My understanding is that when you ask ArcCatalog to show
you "the metadata" for a shapefile, it reads the DBF of the
shapefile and populates the metadata with what's in the
DBF header.  When you ask it to show you the metadata for
a coverage, it reads the items of the PAT and AAT.

But this betrays an important point:  the GIS doesn't know
much about your data.  It knows the names of the attributes,
the data storage type (character, integer, floating), and the
width and for some fields the number of decimal places.  It
knows the numerical extent of the coverage in the units of the
coverage, but it only knows the coordinate system if you've
told it what the coordinate system is.  It knows the number
of points, arcs, and polys (arguably not particularly important
information).  It knows the current name of the coverage,
and the directory path you've used to store the data.
That's it.  And that's not much.  To work with the data in
a semantically meaningful way, we have to add a lot of human
knowledge to both our metadata and our procedures for handling
the data.  The GIS might be able to help us store some of
that information, but it has to come from us, and we have to
keep track of it, at least in the sense of where in the data
store we've put what information.  So in a large sense we
shouldn't expect the GIS to make metadata easy, because the
GIS can only get the most important information from us.

> 3) Are you currently working on modifying your ArcCatalog
> metadata tool?  I was told by Gina to let you know what I
> have been discovering and also to ask you if any one has
> already worked on this problem. If so, I don't want to keep
> trying to find bugs in the system. I am wanting to use
> ArcCatalog's metadata capabilities if it will produce a
> very small amount of errors with mp and gives users an easy
> approach to metadata. Right now I don't know enough about
> how the program works in retrieving the data from the
> coverage and some of the problems that I am having I think
> revolve around that process. I am trying to get this all
> figured out by the end of the month so if you could get
> back to me with any information before then I would
> appreciate it.

If I can chip in my bit here, I'll add that the feature many
of our users need is the ability to paste into the metadata
a commonly-used section like Contact_Information or Source_Citation.
Tkme does this with its concept of snippets, but there's no
similar capability in the ArcCatalog metadata editor.  That
would be a helpful convenience to add.  A simple example is
the snippet of metadata you can get from my web interface to
the USGS directory at http://geology.usgs.gov/tools/metadata/contact.phtml
The results of a search come back as FGDC-style contacts,
which you can copy and paste directly into Tkme.

