Guild home

  1. Layout
  2. Runs
    1. Guild metadata
  3. Resource cache
    1. Resource cache and virtual environments
  4. Runs index

Guild home is a directory that Guild AI uses to store various files.

Every Guild command is associated with a Guild home. If Guild home does not exist, Guild will automatically create it.

By default, Guild home is one of two locations depending on whether or not a command is run in a virtual environment.

If the command is not run in a virtual environment, Guild home is ~/.guild by default. If the command is run in a virtual environment, Guild home is $VIRTUAL_ENV/.guild where $VIRTUAL_ENV is the virtual environment directory.

Guild home may be set explicitly by either defining the environment variable GUILD_HOME or by using the ‑H option when running a Guild command.

Layout

Guild home is structured as follows:

  • Guild homeE.g. ~/.guild or $VIRTUAL_ENV/.guild
  • cache Caches used to optimize various Guild functions
    • resources Cached resources
    • runs Indexed run data
  • runs Active runs
  • trash Deleted runs

Runs

Active runs (i.e. non-deleted runs) are stored in $GUILD_HOME/runs. Each run is stored in a subdirectory with the same name as the run ID.

Run directories contain all files associated with a run, including:

  • Resource source links
  • Output generated during the run
  • Guild metadata

Guild metadata is stored in a subdirectory named .guild in each run directory.

Guild metadata

  • Run directory
  • .guild Guild metadata
    • output Output generated by the run OS process
    • output.index Timestamp and stream information associated with run output
    • attrs Run attributes
      • cmd Run OS process command
      • deps Run dependencies (required resources)
      • env Run OS process environment
      • exit_status Run OS process exit status
      • _extra_NAME Extra config for NAME, defined for the model operation
      • flags Flag values used for the run
      • opref Reference to the operation associated with the run
      • started Started timestamp
      • stopped Stopped timestamp

Resource cache

When Guild resolves resources, it uses GUILD_HOME/cache/resources for two purposes:

  • Location for downloading URL resource sources
  • Location for unpacking archives

Resources are stored in directories that are named using hashes of the corresponding resource source URI, which is an internal representation of a resource source location. If a resource exists in the cache, it is used rather than downloading or unpacking the resource again.

Guild does not remove items from the resource cache.

You may clear the resource cache by deleting GUILD_HOME/cache/resources. Guild commands will continue to function normally and the cache will be repopulated as needed.

Resource cache and virtual environments

By default, Guild reuses the resource cache in ~/.guild/cache/resources (the user-level resource cache) for all virtual environments. This means that cached resources available within virtual environments without having to re-download files. However, it also means that virtual environments can modify the user-level resource cache.

If you want to isolate a virtual environment from the user-level cache, run the following command after you first activate the environment:

guild init --local-resource-cache

Runs index

Guild maintains an index of run data in GUILD_HOME/cache/runs. This index lets Guild lookup values such as training accuracy and loss quickly when performing comparisons.

Guild updates the runs index lazily as it discovers new runs or new data associated with a run.

You may clear the runs cache by deleting GUILD_HOME/cache/runs. Guild commands will continue to function normally and runs will be indexed again as needed.