# 3D Scene Reconstruction (Experimental)

In general, 3D modeling of scenes may allow handling of cases that per-frame projective transformations do not allow, such as registering frames having different viewpoints over a non-flat or specular scene, or recovery of depth from frames having different viewpoints. ALE's implementation of 3D scene reconstruction is considered experimental in versions prior to 0.8.3.

Models

The method of modeling used in 0.8.2 and later is spatial subdivision based on regions of interest, followed by iterative update of the estimated occupancy and color of each region of interest. (For a general discussion of occupancy, see Hans Moravec's work on depth recovery.) In 0.8.1, the method of modeling is decomposition and deformation of a surface. In earlier versions, a per-frame, per-pixel model of depth is used.

Recovered Color and Depth Output

The color and depth output options provide a reconstruction of the scene as viewed from a particular frame's vantage point, or from an arbitrary camera view [0.8.2 and later]. In 0.8.2 and later, normalization of output values by accumulated occupancy can optionally be disabled for unfiltered views. Version 0.8.3 adds support for options --et and --3dpx, which can be useful for 3D cropping applications. --et specifies the minimum accumulated weight required for a pixel to be defined, allowing refinements of crops based on clip planes (`--rc` and `--fc`) established prior to scene reconstruction, whereas --3dpx specifies a region of space to be removed after scene reconstruction. 0.8.3 uses depth image pixels to determine such spatial regions, leaving the pixels undefined, whereas 0.8.4 removes scene geometry prior to rendering, allowing rendering of background objects after the foreground has been removed.

    --3dv <n> <o>     Render, to file <o>, colors as viewed from frame <n>.
    --3dd <n> <o>     Render, to file <o>, depths as viewed from frame <n>.
    --3dvp <args> <o> Render, to file <o>, colors viewed with projective <args>.  [0.8.2 and later]
    --3ddp <args> <o> Render, to file <o>, depths viewed with projective <args>.  [0.8.2 and later]
                        <args> are W H V x y z P Y R:
                             W    image width.
                             H    image height.
                             V    camera view angle.
                             x    translation x component.
                             y    translation y component.
                             z    translation z component.
                             P    rotation around x-axis.
                             Y    rotation around y-axis.
                             R    rotation around z-axis.
    --occ-norm        Normalize output with accumulated occupancy.          [default]     [0.8.2 and later]
    --occ-nonorm      Don't normalize output with accumulated occupancy.                  [0.8.2 and later]
    --et <x>          Set encounter threshold <x> for defined pixels.  [default is 0]     [0.8.3 and later]
    --3dpx <args>     Exclude a specified spatial volume following full-scene             [0.8.3]
                      reconstruction, assuming a single depth at each point
                      in the output.  <args> are:
                             <xmin> <xmax> <ymin> <ymax> <zmin> <zmax>
    --3dpx <args>     Exclude a specified spatial volume following full-scene             [0.8.4 and later]
                      reconstruction.  <args> are:
                             <xmin> <xmax> <ymin> <ymax> <zmin> <zmax>

Filtering [0.8.4 and later]

Version 0.8.4 adds support for filtering color values from the inputs to produce final color renderings. This is now the default. Determining surface depth and gradient for filtering the inputs draws from a set of depth pixels within a disc whose radius can be specified with the --3d-dmr and --3d-fmr options.

Filtering color from views distributed on a disc (about the camera origin) of specified aperture allows simulation of focus effects; such effects can be controlled with the --focus option.

    --3d-filter       Use filtering for 3D color output.                    [default]
    --3d-nofilter     Don't use filtering for 3D color output.
    --3d-dmr <x>      Set radius for filtering median depth to <x>        [default 0]
    --3d-fmr <x>      Set radius for filtering median diff to <x>         [default 0]
    --focus <ft> <op> Create focus region with type <ft> and options <op>:
                         Focus type:
                            d <d>           focus at distance <d>
                            p <x> <y>       focus at point (x, y)
                         Space-separated options may include zero or more of:
                            ci=<ci>  camera index               [default 0]
                            fr=<fr>  focal range                [default 0]
                            ht=<ht>  horizontal tilt gradient   [default 0]
                            vt=<vt>  vertical tilt gradient     [default 0]
                            ap=<ap>  aperture diameter          [default 3]
                            sc=<sc>  sample count               [default 3]
                            sx=<sx>  start x coordinate      [default -Inf]
                            ex=<ex>  end x coordinate         [default Inf]
                            sy=<sy>  start y coordinate      [default -Inf]
                            ey=<ey>  end y coordinate         [default Inf]
                            sd=<sd>  start depth                [default 0]
                            ed=<ed>  end depth                [default Inf]
                            sr=<sr>  view sample randomization:
                                aperture  per aperture            [default]
                                pixel     per pixel
                            fs=<fs>  focal statistic, one of:
                                mean      mean of views           [default]
                                median    median of views

Camera parameter adjustment

In versions 0.8.1 and later, camera parameters can be adjusted according to control points to decrease the error between points in different frames. Perturbation bounds for adjustment can be specified, including an additional upper bound on view angle perturbation, and a stereo threshold can be set for discarding control points after adjustment is complete. An initial camera view angle can also be specified. Versions 0.8.2 and later allow calculation of error by median rather than mean, and also allow disabling of adjustment of orientation and/or position.

    --view-angle <x>  Set the initial diagonal view angle to <x> degrees.
                         (Default is 43.7 degrees or device-specific.)
    --cpp-upper=<x>   Set upper bound <x> for camera parameter perturbation,           [0.8.1 and later]
                      in pixels or degrees [default is 32]
    --cpp-lower=<x>   Set lower bound <x> for camera parameter perturbation,           [0.8.1 and later]
                      in pixels or degrees [default is 0.125]
    --va-upper=<x>    View-angle perturbation upper bound in degrees  [default is 32]  [0.8.1 and later]
    --st <x>          Control point stereo threshold is <x> pixels.    [default is 4]  [0.8.1 and later]
    --cpp-err-mean    Use RMS error to determine camera parameters.         [default]        [0.8.2 and later]
    --cpp-err-median  Use median error to determine camera parameters.                       [0.8.2 and later]
    --vp-adjust       Adjust the view point                                 [default]        [0.8.2 and later]
    --vp-noadjust     Do not adjust the view point                                           [0.8.2 and later]
    --vo-adjust       Adjust the view orientation                           [default]        [0.8.2 and later]
    --vo-noadjust     Do not adjust the view orientation                                     [0.8.2 and later]

Transformation data files [0.8.2 and later]

Transformation data file options allow reading and writing of camera position, orientation, and view angle information.

    --3d-trans-load=x Load initial transformation settings from file x                       [0.8.2 and later]
    --3d-trans-save=x Save final transformation data in file x                               [0.8.2 and later]

Transformation data file format [0.8.2 and later]

The transformation data file uses the following format, starting with a version command, and then describing view angle and euclidean transformation of each frame in turn:

    W 0
    V <view angle>
    E <width> <height> <X> <Y> <Z> <P> <Y> <R>
    V <view angle>
    E <width> <height> <X> <Y> <Z> <P> <Y> <R>
    .
    .
    .
    V <view angle>
    E <width> <height> <X> <Y> <Z> <P> <Y> <R>

Model costs [0.8.1 only]

After the cameras are adjusted, a 2D surface is decomposed and deformed to reduce local cost measures, calculated from angles between neighboring triangles, edge lengths, and differences in color between different views. Multipliers for edge length and angle costs can be specified on the command line.

    --ecm <x>         Set edge length cost multiplier <x>.         [default is 0.001]  [0.8.1 only]
    --acm <x>         Set angle cost multiplier <x>.               [default is 0.001]  [0.8.1 only]

Model rules [0.8.2 and later]

In versions 0.8.2 and later, clip planes can be specified, and a color falloff exponent for pairwise occupancy estimation can be specified. Larger exponents are more forgiving of color differences between pixels in different frames. In versions 0.8.3 and later, error contributions from cameras outside of the generating pair can be assigned non-zero multipliers (`--tcem`), the number of occupancy and color update iterations can be specified (`--oui`), the number of points in space generated for each pixel pair can be specified (`--pa`), and the type of frame pairings can be specified (`--pc`).

    --di-upper <x>    Decimate primary input resolution by at most 2^x [default is 0]  [0.8.4 and later]
    --di-lower <x>    Decimate input resolutions by at least 2^x       [default is 0]  [0.8.4 and later]
    --do-try <x>      Decimate output resolution by 2^x if possible    [default is 0]  [0.8.4 and later]
    --oc              Clip scene to output regions.                                    [0.8.4 and later]
    --no-oc           Do not clip scene to output regions.                  [default]  [0.8.4 and later]
    --fc <x>          Set front-clip to <x> (0 < x < 1)                [default is 0]  [0.8.2 and later]
    --rc <x>          Set rear-clip to <x> (1 < x < inf)             [default is inf]  [0.8.2 and later]
    --fx <x>          Set falloff exponent to <x>                      [default is 0]  [0.8.2 and later]
    --tcem <x>        Set third-camera error multiplier to <x>         [default is 0]  [0.8.3 and later]
    --oui <x>         Set occupancy update iterations to <x>          [default is 10]  [0.8.3 and later]
    --pa <x>          Set pairwise ambiguity to <x>                    [default is 3]  [0.8.3 and later]
    --pc <type>       Set the type of pairwise comparisons:                            [0.8.3 and later]
                         auto     Determine comparisons automatically.      [default]
                         all      Perform all comparisons.