joho

Filename....: JAM-001
Rev.........: 001
Dated.......: 93-07-01
Status .....: Released
Subject.....: JAM message base proposal
Author......: Joaquim Homrighausen
Co-Authors..: Andrew Milner, Mats Birch, Mats Wallin

    ---------------------------------------------------------------------
                                  JAM(mbp)
                The Joaquim-Andrew-Mats Message Base Proposal
    ---------------------------------------------------------------------
            Copyright 1993 Joaquim Homrighausen, Andrew Milner,
                           Mats Birch, Mats Wallin.
                             ALL RIGHTS RESERVED.
    ---------------------------------------------------------------------

    =====================================================================
    Restrictions
    ---------------------------------------------------------------------
    JAM may be used by any developer as long as these specifications are
    followed exactly. JAM may be used free-of-charge by any developer
    for any purpose, commercially or otherwise.

    This document may be freely copied and distributed, but must NEVER be
    distributed in a modified form. If you have an enhancement request,
    please contact the author of this document; do not change it
    yourself.

    All applications that support JAM must include one of the following
    notices in their documentation and somewhere in the product's credit
    section:

    "JAM(mbp) - Copyright 1993 Joaquim Homrighausen, Andrew Milner,
                               Mats Birch, Mats Wallin.
                               ALL RIGHTS RESERVED."

    or

    "This product uses the JAM(mbp) API -
     Copyright 1993 Joaquim Homrighausen, Andrew Milner, Mats Birch,
                    Mats Wallin. ALL RIGHTS RESERVED."

    No organization, company, person, entity, or other being may impose
    any fees for any reason for providing this document or the
    accompanying API. This document and the accompanying API may not be
    sold or otherwise transferred for personal or company gain under any
    circumstances.

    =====================================================================
    Definitions and general notes
    ---------------------------------------------------------------------
    CURRENTREV                1

    JAM                       The JAM message base format.

    CRC                       Cyclic Redundancy Check. All CRC values
                              calculated on strings must assume that the
                              data within the string has been converted
                              to lowercase (A-Z = a-z).

    CRC-32                    32-bit CRC (as used in the Zmodem file
                              transfer protocol) value. The polynom for
                              a CRC-32 is edb88320H and the CRC-32 seed
                              is -1L (ffffffffH).

    uchar                     Unsigned 8-bit value

    ushort                    Unsigned 16-bit value

    ulong                     Unsigned 32-bit value

    UNIX date                 An ulong representing the number of seconds
                              since midnight, January 1, 1970. UNIX-style
                              dates is the only form of time stamps used
                              in JAM (1).

    Message #                 The physical record number within the index
                              file is used as a message number. The
                              lowest message number is one (1) and the
                              highest message number is 4294967295
                              (ffffffffH).

    FTN                       FidoNet Technology Network

    FTS                       FidoNet Technical Standard

    (1) All timestamps created locally (i.e. those not imported from
        other systems) are stored in local time.

    =====================================================================
    Files
    ---------------------------------------------------------------------
    Each conference is made up from four files. How and where these files
    are stored and named is implementation dependant. The only file with
    a fixed minimum size is the .JHR (header data) file. It has a 1024-
    byte block used to hold information about a specific message area as
    described later.

    filename.JHR - Message header data
    filename.JDT - Message text data
    filename.JDX - Message index
    filename.JLR - Lastread information

    A future revision of JAM may also include a file that holds the
    following three items:

      - The highest assigned user number
      - The last generated message ID
      - A global conference list with the conference name, description,
        and physical location of the message base.

    =====================================================================
    .JHR file header
    ---------------------------------------------------------------------
    Below is the format of the 1024-byte record at the beginning of all
    .JHR files. The first actual message header starts at offset 1024 in
    the .JHR file.

    FixedHeaderInfoStruct:
        ulong   Signature;       // <J><A><M> followed by <NUL>
        ulong   datecreated;     // Creation date
        ulong   modcounter;      // Update counter
        ulong   activemsgs;      // Number of active (not deleted) msgs
        ulong   passwordcrc;     // CRC-32 of password to access
        ulong   basemsgnum;      // Lowest message number in index file
        uchar   RESERVED[1000];  // Reserved space
    end;

    MODCOUNTER must be incremented and updated on disk each time an
    application modifies the contents of the message base. When it
    reaches ffffffffH, it wraps to zero.

    ---------------------------------------------------------------------
    BaseMsgNum                        Lowest message number in index file
    ---------------------------------------------------------------------
    This field determines the lowest message number in the index file.
    The value for this field is one (1) when a message area is first
    created. By using this field, a message area can be packed (deleted
    messages are removed) without renumbering it. If BaseMsgNum contains
    500, the first index record points to message number 500.

    BaseMsgNum has to be taken into account when an application
    calculates the next available message number (for creating new
    messages) as well as the highest and lowest message number in a
    message area.

    ---------------------------------------------------------------------
    ????????.JHR                                          Message headers
    ---------------------------------------------------------------------
    The .JHR file contains none or more Header records. Each record
    define one message and contains information about the message and its
    text (if any). The Header record is of variable length. The layout of
    the Header record follows.

    MessageHeader:
        MessageFixedHeader:
            ulong  Signature;    // <J><A><M> followed by <NUL>
            ushort Revision;     // Revision level of header          (1)
            ushort ReservedWord; // Reserved for future use
            ulong  SubfieldLen;  // Length of subfields               (2)
            ulong  TimesRead;    // Number of times message read
            ulong  MSGIDcrc;     // CRC-32 of MSGID line              (3)
            ulong  REPLYcrc;     // CRC-32 of REPLY line              (3)
            ulong  ReplyTo;      // This msg is a reply to..
            ulong  Reply1st;     // First reply to this msg
            ulong  Replynext;    // Next msg in reply chain
            ulong  DateWritten;  // When msg was written
            ulong  DateReceived; // When msg was read by recipient
            ulong  DateProcessed;// When msg was processed by tosser/
                                 // scanner
            ulong  MessageNumber;// Message number (1-based)
            ulong  Attribute;    // Msg attribute, see "Msg Attributes"
            ulong  Attribute2;   // Reserved for future use
            ulong  Offset;       // Offset of text in ????????.JDT file
            ulong  TxtLen;       // Length of message text
            ulong  PasswordCRC;  // CRC-32 of password to access message
            ulong  Cost;         // Cost of message
        end;
        SubField1                // Extra fields as defined below
        .
        .
        SubFieldXX
    end;

    (1) This field is intended for future revisions of the specifications
        to allow the use of a different fixed-length binary message
        header. The current revision level is one (1).

    (2) The SubfieldLen field is set to zero (0) if the header does not
        have any subfield data. I.e. the length of the binary header is
        not included in this field.

    (3) When calculating the CRC-32 of the MSGID and REPLY lines, the
        text ^aMSGID: and ^aREPLY: should be removed as well as all
        leading and trailing white space characters.

    The SubField structure is made up of an ID, a length specifier, and
    a block of data. Zero or more subfields may follow the fixed-length
    binary header. SubFields are not stored in any specific order and
    are not terminated by any specific character unless otherwise
    specified.

    SubField:
        ushort  LoID;            // Field ID, 0-65535
        ushort  HiID;            // Reserved for future use
        ulong   datlen;          // Length of buffer that follows
        uchar   Buffer[datlen];  // DATLEN bytes of data
    end;

    ---------------------------------------------------------------------
    Defined LoID codes
    ---------------------------------------------------------------------

    ID=0, Name=OADDRESS

    A network address. This is used to specify the originating address.
    More than one OADDRESS field may exist. DATLEN must not exceed 100
    characters. For a FidoNet-style address, this field must follow the
    ZONE:NET/NODE.POINT@DOMAIN format where .POINT is excluded if zero
    and @DOMAIN is excluded if unknown.

    ID=1, Name=DADDRESS

    A network address. This is used to specify the destination address.
    More than one DADDRESS field may exist (e.g. carbon copies). DATLEN
    must not exceed 100 characters. For a FidoNet-style address, this
    field must follow the ZONE:NET/NODE.POINT@DOMAIN format where .POINT
    is excluded if zero and @DOMAIN is excluded if unknown.

    ID=2, Name=SENDERNAME

    The sender (author) of the message. DATLEN must not exceed 100
    characters.

    ID=3, Name=RECEIVERNAME

    The recipient of the message. DATLEN must not exceed 100 characters.

    ID=4, Name=MSGID

    Used to store the message identification data. All data not relevant
    to the actual ID string, including leading and trailing white space
    characters should be removed. DATLEN must not exceed 100 characters.

    ID=5, Name=REPLYID

    Used to store the message reply data. All data not relevant to the
    actual reply string, including leading and trailing white space
    characters should be removed. DATLEN must not exceed 100 characters.

    ID=6, Name=SUBJECT

    The subject of the message. DATLEN must not exceed 100 characters.
    Note that this field may not be used for FidoNet-style file attaches
    or file requests.

    ID=7, Name=PID

    Used to store the FTN PID kludge line. Only the actual PID data is
    stored and ^aPID: is stripped along with any leading and trailing
    white space characters. DATLEN must not exceed 40 characters.

    ID=8, Name=TRACE

    This is also referred to as ^aVia information in FTNs. It contains
    information about a system which the message has travelled through.
    The format of the field is  where:

       YYYY is the year (1992-9999)
         MM is the month (01-12)
         DD is the day (01-31)
         HH is the hour (00-23)
         MM is the minute (00-59)
         SS is the second (00-59)

    The timestamp is stored in ASCII (0-9) characters. The network
    address is the address of the system. It is expressed in ASCII
    notation in the native format of the forwarding system.

    ID=9, Name=ENCLOSEDFILE

    A file attached to the message. Only one filename may be specified
    per subfield. No wildcard characters are allowed. If this subfield
    is present in a message header, the ATTRIBUTE must include the
    MSG_FILEATTACH bit.

    ID=10, Name=ENCLOSEDFILEWALIAS

    Identical to ENCLOSEDFILE with the exception that the filename is
    followed by a  (00H) and an alias filename to be transmited to
    the remote system in place of the local name of the file.

    ID=11, Name=ENCLOSEDFREQ

    A request for one or more files. Only one filemask may be specified
    per subfield. If the filemask contains a complete path, it is to be
    regarded as an update file request. If this subfield is present in a
    message header, the ATTRIBUTE must include the MSG_FILEREQUEST bit.
    To indicate that a password is to be transmitted along with the
    request, a  (00H) character followed by the password is
    appended. E.g. SECRET*.*MYPASSWORD.

    ID=12, Name=ENCLOSEDFILEWCARD

    One or more files attached to the message. Only one filename may be
    specified per subfield. Wildcard characters are allowed. If this
    subfield is present in a message header, the ATTRIBUTE must include
    the MSG_FILEATTACH bit.

    ID=13, Name=ENCLOSEDINDIRECTFILE

    One or more files attached to the message. The filename points to an
    ASCII file with one filename entry per line. If alias filenames are
    to be used, they are specified after the actual filename and
    separated by a  (00H) character, e.g. C:\MYFILE.LZHNEWS.
    Wildcard characters are not allowed.

    ID=1000, Name=EMBINDAT

    Reserved for future use.

    ID=2000, Name=FTSKLUDGE

    An FTS-compliant "kludge" line not otherwise represented here. All
    data not relevant to the actual kludge line, including leading and
    trailing white space and ^A (01H) characters should be removed.
    DATLEN must not exceed 255 characters. The FTS kludges INTL, TOPT,
    and FMPT must never be stored as separate SubFields. Their data must
    be extracted and used for the address SubFields.

    ID=2001, Name=SEENBY2D

    Used to store two-dimensional (net/node) SEEN-BY information often
    used in FTN conference environments. Only the actual SEEN-BY data is
    stored and ^aSEEN-BY: or SEEN-BY: is stripped along with any leading
    and trailing white space characters.

    ID=2002, Name=PATH2D

    Used to store two-dimensional (net/node) PATH information often used
    in FTN conference environments. Only the actual PATH data is stored
    and ^aPATH: is stripped along with any leading and trailing white
    space characters.

    ID=2003, Name=FLAGS

    Used to store the FTN FLAGS kludge information. Note that all FLAG
    options that have binary representation in the JAM message header
    must be removed from the FLAGS string prior to storing it. Only
    the actual flags option string is stored and ^aFLAGS is stripped
    along with any leading and trailing white space characters.

    ID=2004, Name=TZUTCINFO

    Time zone information. This subfield consists of four mandatory
    bytes and one optional. The first character may be a plus (+) or a
    minus (-) character to indicate a location east (plus) or west
    (minus) of UTC 0000. The plus character is implied unless the first
    character is a minus character. The following four bytes must be
    digits in the range zero through nine and indicates the offset in
    hours and minutes. E.g. 0100 indicates an offset of one hour east of
    UTC.

    ---------------------------------------------------------------------
    Message attributes
    ---------------------------------------------------------------------
    MSG_LOCAL       (0x00000001L)   // Msg created locally
    MSG_INTRANSIT   (0x00000002L)   // Msg is in-transit
    MSG_PRIVATE     (0x00000004L)   // Private
    MSG_READ        (0x00000008L)   // Read by addressee
    MSG_SENT        (0x00000010L)   // Sent to remote
    MSG_KILLSENT    (0x00000020L)   // Kill when sent
    MSG_ARCHIVESENT (0x00000040L)   // Archive when sent
    MSG_HOLD        (0x00000080L)   // Hold for pick-up
    MSG_CRASH       (0x00000100L)   // Crash
    MSG_IMMEDIATE   (0x00000200L)   // Send Msg now, ignore restrictions
    MSG_DIRECT      (0x00000400L)   // Send directly to destination
    MSG_GATE        (0x00000800L)   // Send via gateway
    MSG_FILEREQUEST (0x00001000L)   // File request
    MSG_FILEATTACH  (0x00002000L)   // File(s) attached to Msg
    MSG_TRUNCFILE   (0x00004000L)   // Truncate file(s) when sent
    MSG_KILLFILE    (0x00008000L)   // Delete file(s) when sent
    MSG_RECEIPTREQ  (0x00010000L)   // Return receipt requested
    MSG_CONFIRMREQ  (0x00020000L)   // Confirmation receipt requested
    MSG_ORPHAN      (0x00040000L)   // Unknown destination
    MSG_ENCRYPT     (0x00080000L)   // Msg text is encrypted          (1)
    MSG_COMPRESS    (0x00100000L)   // Msg text is compressed         (1)
    MSG_ESCAPED     (0x00200000L)   // Msg text is seven bit ASCII    (1)
    MSG_FPU         (0x00400000L)   // Force pickup
    MSG_TYPELOCAL   (0x00800000L)   // Msg is for local use only
    MSG_TYPEECHO    (0x01000000L)   // Msg is for conference distribution
    MSG_TYPENET     (0x02000000L)   // Msg is direct network mail
    MSG_NODISP      (0x20000000L)   // Msg may not be displayed to user
    MSG_LOCKED      (0x40000000L)   // Msg is locked, no editing possible
    MSG_DELETED     (0x80000000L)   // Msg is deleted

    (1) This revision of JAM does not include compression, encryption, or
        escaping. The bits are reserved for future use.

    =====================================================================
    ????????.JDT                                             Message text
    ---------------------------------------------------------------------
    The .JDT file contains the text of messages. The text is stored as an
    stream of seven or eight bit ASCII data. Allowed characters in the
    text are 00H through ffH unless the header ATTRIBUTE field has the
    MSG_ESCAPED bit enabled, in which case the legal range of data is 20H
    through 7eH.

    An escaped character is stored as \ where  is the two digit
    hexadecimal ASCII value of the character. A single \ is stored as \\
    or \5C. The case of the hexadecimal ASCII value is irrelevant, i.e.
    5c is treated as 5C.

    =====================================================================
    ????????.JDX                                            Message index
    ---------------------------------------------------------------------
    The .JDX file is used to quickly locate messages for any given user
    name or to locate a message with a specific number. Each record in
    the file consists of two ulongs. The first ulong holds the CRC-32 of
    the recipient's name (lowercase), the second ulong holds the
    physical offset of the message header in the .JHR (header) file.

    The record number (+BaseMsgNum) within the .JDX file determines a
    message's number.

    If both ulongs are -1 (ffffffffH), there is no corresponding message
    header.

    =====================================================================
    ????????.JLR                                         Lastread storage
    ---------------------------------------------------------------------
    The .JLR file is used to maintain a user's position within a message
    area. The layout of the "lastread" record follows. One record per
    user is required.

    LastRead:
        ulong   UserCRC;         // CRC-32 of user name (lowercase)   (1)
        ulong   UserID;          // Unique UserID
        ulong   LastReadMsg;     // Last read message number
        ulong   HighReadMsg;     // Highest read message number
    end;

    (1) The functions to convert a string to lowercase characters that
        are provided in the API will only convert characters A-Z (into
        a-z). It is required that this convention is followed by all
        applications.

    The UserID field is a unique number for each user. If the "lastread"
    record is deleted, UserCRC and UserID are both set to -1
    (ffffffffH). An application may not depend on any specific order in
    the .JLR file. A user's "lastread" record may appear anywhere in the
    file and must be searched for when retrieving it and when storing an
    updated record.

    =====================================================================
    Updating message headers
    ---------------------------------------------------------------------
    If a header record grows after is has been retrieved from the .JHR
    file, it must be appended to the end of the .JHR file since it would
    overwrite the following header otherwise. The .JDX file must be
    properly updated to indicate the new location of the header record.
    The old header record must be changed to indicate that it has been
    deleted by setting the MSG_DELETED bit in the Attribute field and the
    TextLen field to zero (to prevent a maintenance program from removing
    the message text that is now pointed to by another header).

    =====================================================================
    Message base sharing and locking
    ---------------------------------------------------------------------
    To allow several programs to access the message base at any given
    time, region locking is used to protect the message base from being
    corrupted during updates.

    When an application needs to write to any of the message base files,
    it must first attempt to lock the first byte of the .JHR (header)
    file. If the lock call fails, the application must either fail or
    attempt to lock the file again. The message base files may under no
    circumstances be updated if the application cannot successfully lock
    the .JHR file.

    Note that data acquired (read) from the message base may not be used
    when writing data to the message base, unless the application has
    maintained a lock of the message base from the time the data was
    acquired or the MODCOUNTER field is the same as when the data was
    acquired.

    The application must open the files in shareable (DENYNONE) read/
    write or readonly mode. The only exception to this is an application
    that requires exclusive access to the message base, such as a message
    base maintenance utility, it should open the files in non-shareable
    (DENYALL) read/write mode.

    =====================================================================
    Reply threads and linking
    ---------------------------------------------------------------------
    JAM introduces a new reply link pointer, not commonly used today.
    This section is an attempt to describe how reply threads, reply
    linking, and this new reply link pointer is implemented in JAM.

    One of the major differences is that reply threads in JAM are not
    based on similar or identical subjects of messages since this method
    does not allow for proper reply threads.

    The method used in JAM is based on the immediate relation between any
    given message and direct replies to it. This is supported by many
    message editors by using the MSGID and REPLY FTS kludge fields. These
    are common, although expressed differently, in messages not based on
    FidoNet technology, such as RFC-822. The obvious advantages include
    allowing a program to easily find the original message to a reply,
    and to find all replies to any given message.

    The reply thread information consists of three fields: ReplyTo,
    Reply1st, and ReplyNext. The reason for three fields, as opposed to
    just two, is that with two fields, it is only possible to keep track
    of the original message of a reply (which is sufficient) and one
    reply to any given message (which is not sufficient). With three
    fields, it is possible to maintain a thread of any number of replies
    to any given message.

    In the description of the different fields below, the following
    messages and message numbers will be referred to:

      1 -> 2 -> 4 -> 5
      :    :
      :    +--> 8
      :
      +--> 3 -> 7
      :
      +--> 6

    Message number two, three, and six are replies to message number one.
    Message number four and eight are replies to message number two.
    Message number seven is a reply to message number three.
    Message number five is a reply to message number four.

    ---------------------------------------------------------------------
    ReplyTo
    ---------------------------------------------------------------------
    This field holds the number of the message that this message is a
    reply to. In the example above, the ReplyTo field would contain the
    following values:

    Message number one would contain zero; message number two, three, and
    six, would contain one; message number four and eight would contain
    two; message number seven would contain three, and message number
    five would contain four.

    ---------------------------------------------------------------------
    Reply1st
    ---------------------------------------------------------------------
    This field holds the number of the first message that is a reply to
    this message. In the example above, the Reply1st field would contain
    the following values:

    Message number one would contain two, message number three would
    contain seven, and message number four would contain five. All other
    messages would contain zero.

    ---------------------------------------------------------------------
    ReplyNext
    ---------------------------------------------------------------------
    This field is used to create the actual message thread or chain. In
    the event that there is more than one reply to any given message, it
    is necessary to maintain a thread of all the replies; this is due to
    the fact that the original message can only hold information about
    the first reply (the Reply1st field) to it.

    The first reply (which the original message's Reply1st field holds),
    has its ReplyNext field pointing to the second reply, the second
    reply's ReplyNext field poinst to the third reply, and so on.

    In the example above, the ReplyNext field would contain the following
    values:

    Message number two would contain three, message number three would
    contain six, and message number four would contain eight. All other
    messages would contain zero.

    =====================================================================
    Contacts
    ---------------------------------------------------------------------
    Joaquim Homrighausen

    Andrew Milner

    Mats Wallin

    // end of file "jam.doc"