%% -*- mode: LaTeX -*-
%%
%% Copyright (c) 1999 The University of Utah and the Computer Systems
%% Laboratory at the University of Utah (CSL).
%%
%% This file is part of Flick, the Flexible IDL Compiler Kit.
%%
%% Flick is free software; you can redistribute it and/or modify it under the
%% terms of the GNU General Public License as published by the Free Software
%% Foundation; either version 2 of the License, or (at your option) any later
%% version.
%%
%% Flick is distributed in the hope that it will be useful, but WITHOUT ANY
%% WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
%% FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
%% details.
%%
%% You should have received a copy of the GNU General Public License along with
%% Flick; see the file COPYING.  If not, write to the Free Software Foundation,
%% 59 Temple Place #330, Boston, MA 02111, USA.
%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Metadata (META) Representation}
\label{cha:META}

To keep track of attributes that are associated with Flick's intermediate data
structures, Flick stores \emph{metadata} in the intermediate files that it
creates (i.e., \AOI{} and \PRESC{} files).  Metadata allows Flick to track the
sources and sinks (destinations) of objects, to describe relationships among
those sources and sinks, to attach additional semantics or classifications to
data, and so on.  All of this information is stored in a format called
\META{}\@.

Metadata is used to classify objects in the intermediate files so that Flick
can partition its output in various ways.  For example, Flick may put certain
types of data in one file and other types of data in another.  Alternately,
Flick might completely suppress the output of certain kinds of data.  The
\META{} format is currently constrained to representing relatively simple
attributes --- e.g., the input files associated with various objects --- but
the \META{} language may be extended in the future to support more complex
kinds of annotations.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{META Overview}
\label{sec:META:META Overview}

The \META{} data structures are defined in \filename{mom/meta.x}, and the
library for manipulating \META{} data is contained in the directory
\filename{libmeta}.

\META{} provides two basic abstractions: \ctype{io_file}s, representing input
files,\footnote{\ctype{io_file}s are not currently used to represent output
files.} and \ctype{data_channel}s, representing separate ``streams'' of data
within a single file.  These structures are used to classify data according to
their source and purpose.  An \ctype{io_file} is referenced by data structures
that ``arise'' from the input file represented by the \ctype{io_file} object.
A \ctype{data_channel} is used to classify the results generated by processing
the input definitions.  For example, multiple channels could be used to
separate the client-side and server-side C declarations that result from a
single \IDL{} definition.

In addition to the basic \ctype{io_file} and \ctype{data_channel} structures,
\META{} defines structures for describing sets of files and channels.  Sets are
described by \emph{masks}: patterns that may be used to find files and channels
of interest.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{META Data Structures}
\label{sec:META:META Data Structures}

The primary metadata objects are contained within arrays which are collected
under a root object of type \ctype{meta}:

\begin{verbatim}
struct meta {
        io_file         files<>;
        data_channel    channels<>;
};
\end{verbatim}

\noindent A reference to an \ctype{io_file} or \ctype{data_channel} is
implemented as an integer index into the appropriate array within a
\ctype{meta} structure.

\begin{verbatim}
typedef int io_file_index;
typedef int data_channel_index;
\end{verbatim}

\noindent Other \META{} data structures, such as descriptions of file and
channel sets, are not stored in the root \ctype{meta} structure but are instead
stored where they are used.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Files}
\label{subsec:META:Files}

Files are tracked with the \ctype{io_file} structure:

\begin{verbatim}
struct io_file {
        string          id<>;           /* Filename */
        unsigned int    flags;          /* Holds the above flags */
        int             references;     /*
                                         * How many other files have included
                                         * this one
                                         */
        io_file_index   includes<>;     /* References to included files */
};
\end{verbatim}

\noindent The \cidentifier{flags} field contains a set of the following values:

\begin{cidentifierlist}
  \item[IO_FILE_INPUT] Indicates that the \ctype{io_file} represents an input
  file.

  \item[IO_FILE_OUTPUT] Indicates that the \ctype{io_file} represents an output
  file.  \emph{This flag is not currently used by Flick.}

  \item[IO_FILE_BUILTIN] Indicates that the \ctype{io_file} does not correspond
  to any actual file.  Rather, data from this ``file'' is already built into
  Flick.

  \item[IO_FILE_ROOT] Indicates that this \ctype{io_file} represents the root
  \IDL{} file from which all others are \cidentifier{#include}d.

  \item[IO_FILE_SYSTEM] Indicates that this \ctype{io_file} represents a system
  file (i.e., a system header file, generally \cidentifier{#include}d with
  \cidentifier{<>}'s).
\end{cidentifierlist}

The main set of \ctype{io_file} objects is generated by a Flick front end when
it processes a root \IDL{} file.  In addition, the front end will create any
``builtin'' files that it needs to indicate definitions that are known a~priori
to the front end.  For example, the \CORBA{} front end (described in
Section~\ref{sec:FE:CORBA Front End}) creates a builtin \ctype{io_file} to
represent the source of the predefined \idl{CORBA::Object} interface type.  The
set of \ctype{io_file}s created by a front end encodes just about everything
that Flick cares about from the files, from their names to their place(s) in
the inclusion graph.

%% \subsubsection{Selecting File Sets}

% Information about the file hierarchy is nice, but unless we're able to use
% this information, it is relatively useless.  So, to aid in the processing of
% io_files, the io_file_mask auxiliary structure was created with the ability
% to select sets of io_files.

The \ctype{io_file_mask} structure describes a set of \ctype{io_file} objects:

\begin{verbatim}
struct io_file_mask {
        unsigned int    mask_flags;     /* Flags specific to the mask */
        string          id<>;           /* An identifier to match */
        unsigned int    set_flags;      /*
                                         * Match when all of these flags are
                                         * set in the io_file
                                         */
        unsigned int    unset_flags;    /*
                                         * Match when all of these flags aren't
                                         * set in the io_file
                                         */
};
\end{verbatim}

This structure is supposed to be somewhat opaque; \filename{libmeta} provides
several functions for dealing with them.  The construction of
\ctype{io_file_mask}s is handled by a tag list function (see
Section~\ref{subsec:Coding:Tags}) that sets all the fields and flags based on
the tags passed in.  For example, the following statement constructs an
\ctype{io_file_mask} that matches only system files:

\begin{verbatim}
        meta_make_file_mask(FMA_SetFlags, IO_FILE_SYSTEM,
                            FMA_TAG_DONE);
\end{verbatim}

\noindent The \cfunction{meta_make_file_mask} function is fully described in
Section~\ref{sec:META:The META Library}, below.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Channels}
\label{subsec:META:Channels}

Channels are described by the \ctype{data_channel} structure:

\begin{verbatim}
struct data_channel {
        io_file_index           input;          /* input for the channel */
        string                  id<>;           /* semantic id */
        unsigned int            flags;          /* Holds the above flags */
        io_file_index           outputs<>;      /*
                                                 * List of output files for
                                                 * the channel (unused)
                                                 */
};
\end{verbatim}

\noindent Note that the \cidentifier{output} field is currently unused, because
\ctype{io_file}s are currently used only for input files.  The possible flags
within the \cidentifier{flags} field are these:

\begin{cidentifierlist}
  \item[DATA_CHANNEL_SQUELCHED] Indicates that the channel is squelched:
  nothing on this channel should be output.

  \item[DATA_CHANNEL_DECL] Indicates that anything on this channel is a
  declaration.

  \item[DATA_CHANNEL_IMPL] Indicates that anything on this channel is a
  definition or implementation.
\end{cidentifierlist}

The purpose of a channel is to describe a data path for some object, thus
allowing Flick to classify it and determine its origin and destination.  This
knowledge is primarily useful for squelching \IDL{} definitions that should not
appear in the output.  For example, a Flick back end may need to suppress
definitions and/or declarations that arise from \IDL{} system header files: the
definitions of system objects may be built into the targeted \RPC{}/\RMI{}
runtime library.  Similarly, a user might ask Flick to suppress definitions
that come from \cidentifier{#include}d \IDL{} files, but to keep the
corresponding declarations.  This is useful when the included \IDL{} files are
intended to be compiled separately.

%% \subsubsection{Selecting Channel Sets}

Similar to \ctype{io_file}s, channels can be selected with the help of a mask
structure:

\begin{verbatim}
struct data_channel_mask {
        unsigned int    mask_flags;     /* Flags for the mask */
        io_file_mask    *input;         /* Input file mask */
        string          id<>;           /* ID to match on */
        unsigned int    set_flags;      /*
                                         * Match when all of these flags are
                                         * set in the io_file
                                         */
        unsigned int    unset_flags;    /*
                                         * Match when all of these flags aren't
                                         * set in the io_file
                                         */
};
\end{verbatim}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The META Library}
\label{sec:META:The META Library}

The \META{} library contains a relatively small number of functions of creating
and manipulating \META{} data structures.  The source code for the library is
in the \filename{libmeta} directory, and the library header file is
\filename{mom/libmeta.h}.  The various \META{} library functions are summarized
below.

\begin{cprototypelist}
  \item[void init_meta(meta *m)]
  %
  Initialize the given \ctype{meta} structure.  The structure is set to contain
  no files or channels.

  \item[void check_meta(meta *m)]
  %
  Check that the given \ctype{meta} structure is well-formed.  This function
  uses \cfunction{assert} to verify the contents of the structure.

  \item[void print_meta(meta *m, FILE *file, int indent)]
  %
  Pretty-print the given \ctype{meta} structure to the given file, using the
  specified initial indentation value.
\end{cprototypelist}

%%
%%

\noindent The following functions are useful for manipulating files:

\begin{cprototypelist}
  \item[io_file_index meta_add_file(meta *m, const char *id, int flags)]
  %
  If the given \ctype{meta} contains an \ctype{io_file} with the specified
  \cidentifier{id} and \cidentifier{flags}, return the index of that file.
  Otherwise, create a new \ctype{io_file} within \cidentifier{m}, initialize
  it, and return the index of the newly created file.

  \item[io_file_index meta_find_file(meta *m, const char *id, int flags,
                                     int absolute_path)]
  %
  Find a file in the given \ctype{meta} structure.  If \cidentifier{id} is
  null, then match only the \cidentifier{flags}; similarly, if
  \cidentifier{flags} is zero, match only on \cidentifier{id}.  The
  \cidentifier{absolute_path} argument determines whether \cidentifier{id} is
  an absolute file name or just the file part (i.e., non-directory part) of the
  name.  Return the index of the located \ctype{io_file} or \cliteral{-1} if no
  such file exists.

  \item[void meta_include_file(meta *m, io_file_index file,
                               io_file_index included_file)]
  %
  Record the fact that the file at index \cidentifier{file} includes the file
  at index \cidentifier{included_file}.
  %% Irrelevant comment:
  %% Note that a single file can be included more than once.

  \item[void meta_check_file(meta *m, io_file_index file)]
  %
  Check the integrity of the \ctype{io_file} at the given index in
  \cidentifier{m}.  Errors are reported as \ctype{assert} failures (i.e., core
  dumps).

  \item[void meta_print_file(meta *m, FILE *file, int indent,
                             io_file_index idx)]
  %
  Pretty-print the \ctype{io_file} at the given index.  The initial indentation
  level is specified by \cidentifier{indent}.

  \item[io_file_mask meta_make_file_mask(int tag, ...)]
  %
  Create and return an \ctype{io_file_mask} structure, initialized according to
  the tag list given in the function arguments.  (Tag list functions are
  described in Section~\ref{subsec:Coding:Tags}.)  A mask defines a \emph{match
  set} of \ctype{io_file} patterns; a particular \ctype{io_file} can be
  compared to an \ctype{io_file_mask} to determine if the \ctype{io_file} meets
  the criteria specified by the mask, and is therefore a member of the mask's
  match set.

  The tags and associated tag values for the \cfunction{meta_make_file_mask}
  function are these:
  %
  \begin{cidentifierlist}
    \item[FMA_TAG_DONE]
    %
    \tagtypenull{} Marks the end of the tag list.

    \item[FMA_MatchesID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{include}
    \ctype{io_file}s whose complete file names match the given string.
    Matching is done with \cfunction{strcmp} (i.e., exact string matching).

    \item[FMA_ExcludesID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{exclude}
    \ctype{io_file}s whose complete file names match the given string.

    \item[FMA_MatchesDirID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{include}
    \ctype{io_file}s containing file names with directory portions that match
    the given string.

    \item[FMA_ExcludesDirID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{exclude}
    \ctype{io_file}s containing file names with directory portions that match
    the given string.

    \item[FMA_MatchesFileID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{include}
    \ctype{io_file}s containing file names with file (non-directory) portions
    that match the given string.

    \item[FMA_ExcludesFileID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{exclude}
    \ctype{io_file}s containing file names with file (non-directory) portions
    that match the given string.

    \item[FMA_SetFlags]
    %
    \tagtype{\ctype{unsigned int}} Specifies that the match set should include
    \ctype{io_file}s in which the specified flags are set.  Additional flags
    \emph{may} be set in the \ctype{io_file}, but only the specified flags are
    \emph{required} to be set.

    \item[FMA_UnsetFlags]
    %
    \tagtype{\ctype{unsigned int}} Specifies that the match set should include
    \ctype{io_file}s in which the specified flags are unset.
  \end{cidentifierlist}
  %
  An \ctype{io_file_mask} can specify at most one file name matching function:
  in other words, do not specify more than one of the above
  ``\cidentifier{ID}'' tags when creating a mask.  In contrast, an
  \ctype{io_file_mask} can specify matching against both set and unset flags.

  \item[int meta_match_file_mask(meta *m, io_file_mask *ifm,
                                 io_file_index file)]
  %
  Compare the specified \ctype{io_file} (i.e., the file at index
  \cidentifier{file} in \cidentifier{m}) against the given
  \ctype{io_file_mask}.  Return a non-zero value if the file matches the mask
  --- meaning that the file is a member of the mask's match set --- and zero if
  it does not match.

  \item[void meta_squelch_file(meta *m, io_file_index file,
                               data_channel_mask *dcm)]
  %
  Squelch a set of data channels that come from the specified file \emph{and
  from all files that are included by that file}.  The squelched channels are
  those that match \cidentifier{dcm}.  When a channel is ``squelched,'' Flick
  will not produce output from data structures that refer to that channel.

  \item[void meta_squelch_files(meta *m, io_file_mask *ifm,
                                data_channel_mask *dcm)]
  %
  Apply \cfunction{meta_squelch_file} to all of the files in \cidentifier{m}
  that match \cidentifier{ifm}.  The squelched channels are those that match
  \cidentifier{dcm}.

  \item[void meta_print_file_mask(FILE *file, int indent, io_file_mask *ifm)]
  %
  Pretty-print the given \ctype{io_file_mask}.
\end{cprototypelist}

%%
%%

\noindent The following functions are useful for manipulating files:

\begin{cprototypelist}
  \item[data_channel_index meta_add_channel(meta *m, io_file_index input,
                                            const char *id)]
  %
  Create a new \ctype{data_channel} within \cidentifier{m}.  The new channel
  has the given \cidentifier{id} and refers to the indicated \ctype{io_file}.
  Return the index of the newly created channel.

  \item[data_channel_index meta_find_channel(meta *m, io_file_index input,
                                             const char *id, int flags)]
  %
  Find a channel in the given \ctype{meta} structure with the given
  attributes, and return the index of the located channel.  If no such channel
  exists, return \cliteral{-1}.

  \item[void meta_add_channel_output(meta *m, data_channel_index channel,
                                     io_file_index output)]
  %
  Add an output file to the specified channel.  \emph{Because \ctype{io_file}s
  are currently used only for input files, this function is unused.}

  \item[void meta_check_channel(meta *m, data_channel_index channel)]
  %
  Check the integrity of the \ctype{data_channel} at the given index in
  \cidentifier{m}.  Errors are reported as \ctype{assert} failures (i.e., core
  dumps).

  \item[void meta_print_channel(meta *m, FILE *file, int indent,
                                data_channel_index channel)]
  %
  Pretty-print the \ctype{data_channel} at the given index.  The initial
  indentation level is specified by \cidentifier{indent}.

  \item[data_channel_mask meta_make_channel_mask(int tag, ...)]
  %
  Create and return a \ctype{data_channel} structure, initialized according to
  the tag list given in the function arguments.  A mask defines a \emph{match
  set} of \ctype{data_channel} patterns; a particular \ctype{data_channel} can
  be compared to a \ctype{data_channel_mask} to determine if the
  \ctype{data_channel} meets the criteria specified by the mask, and is
  therefore a member of the mask's match set.

  The tags and associated tag values for the \cfunction{meta_make_channel_mask}
  function are these:
  %
  \begin{cidentifierlist}
    \item[CMA_TAG_DONE]
    %
    \tagtypenull{} Marks the end of the tag list.

    \item[CMA_MatchesInput]
    %
    \tagtype{\ctype{io_file_mask}} Specifies that the match set should
    \emph{include} \ctype{data_channel}s that refer to input files matching the
    specified \ctype{io_file_mask}.

    \item[CMA_ExcludesInput]
    %
    \tagtype{\ctype{io_file_mask}} Specifies that the match set should
    \emph{exclude} \ctype{data_channel}s that refer to input files matching the
    specified \ctype{io_file_mask}.

    \item[CMA_MatchesID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{include}
    \ctype{data_channel}s whose \cidentifier{id} field matches the given
    string.  Matching is done with \cfunction{strcmp} (i.e., exact string
    matching).

    \item[CMA_ExcludesID]
    %
    \tagtype{\ctype{char *}} Specifies that the match set should \emph{exclude}
    \ctype{data_channel}s whose \cidentifier{id} field matches the given
    string.

    \item[CMA_SetFlags]
    %
    \tagtype{\ctype{unsigned int}} Specifies that the match set should include
    \ctype{data_channel}s in which the specified flags are set.  Additional
    flags \emph{may} be set in the \ctype{data_channel}, but only the specified
    flags are \emph{required} to be set.

    \item[CMA_UnsetFlags]
    %
    \tagtype{\ctype{unsigned int}} Specifies that the match set should include
    \ctype{data_channel}s in which the specified flags are unset.
  \end{cidentifierlist}

  \item[int meta_match_channel_mask(meta *m, data_channel_mask *dcm,
                                    data_channel_index channel)]
  %
  Compare the specified \ctype{data_channel} (i.e., the channel at index
  \cidentifier{channel} in \cidentifier{m}) against the given
  \ctype{data_channel_mask}.  Return a non-zero value if the channel matches
  the mask --- i.e., is in the mask's match set --- and zero if it does not.

  \item[void meta_squelch_channel(meta *m, data_channel_index channel)]
  %
  Mark the indicated channel as squelched.  When a channel is ``squelched,''
  Flick will not produce output from data structures that refer to that
  channel.

  \item[void meta_squelch_channels(meta *m, data_channel_mask mask)]
  %
  Squelch all of the channels in \cidentifier{m} that match the given
  \ctype{data_channel_mask}.

  \item[void meta_print_channel_mask(FILE *file, int indent,
                                     data_channel_mask dcm)]
  %
  Pretty-print the given \ctype{data_channel_mask}.
\end{cprototypelist}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Summary and Comments}
\label{sec:META:Summary and Comments}

In retrospect, it is apparent that the concept of a \ctype{data_channel} could
be generalized to encompass the abilities of an \ctype{io_file} structure.
Currently, the only real difference between the two structures is that a file
can have multiple inputs --- the included files --- but that a channel can have
only one.  By allowing channels to have multiple inputs, Flick might be able to
express the inclusion file graph as well as other dependencies between
channels.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% End of file.

