PR #29564 [AutoDiff] Fix undiagnosed active `modify` accessor applications.

dan-zheng5 years ago (edited 5 years ago)

Semantically, yielded addresses of modify accessor applications can be viewed
as projections into inout arguments.

Activity analysis should reflect this assumption: yielded addresses should
propagate variedness and usefulness to inout arguments of modify accessor
applications.

Now, previously undetected active modify accessor applications are diagnosed.

Resolves TF-1115.
TF-1080 tracks modify accessor differentiation support.

Example modify accessor application:

array[index] = x

// function_ref Array.subscript.modify
%fn = function_ref @$sSayxSiciM :
    $@yield_once @convention(method) <τ_0_0> (Int, @inout Array<τ_0_0>) -> @yields @inout τ_0_0
(%elt_addr, %token) = begin_apply %fn<Float>(%index, %array) :
    $@yield_once @convention(method) <τ_0_0> (Int, @inout Array<τ_0_0>) -> @yields @inout τ_0_0
store %x to [trivial] %elt_addr : $*Float

If %array is active, %x should also be active.
Before this patch, active %array did not always imply active %x.

TF-1115 example (undiagnosed active modify accessor application):

@differentiable
func TF_1115(x: Float) -> Float {
    var array: [Float] = [0]
    array[0] = x
    return array[0]
}
print(valueWithGradient(at: 4, in: TF_1115))

Before:

$ swift tf-1115.swift
# No error, but incorrect zero derivative.
(value: 4.0, gradient: 0.0)

After:

$ swift tf-1115.swift
tf-1115.swift:1:2: error: function is not differentiable
@differentiable
~^~~~~~~~~~~~~~
tf-1115.swift:2:6: note: when differentiating this function definition
func TF_1115(x: Float) -> Float {
     ^
tf-1115.swift:4:14: note: differentiation of coroutine calls is not yet supported
    array[0] = x
             ^

Note: the assumption that yielded addresses are always a projection into
the inout argument is a safe over-approximation but not always true.

In practice, this should be fine for now.

Example:

var global: Float = 1
extension Float {
  var projection: Float {
    get { self }
    // This `modify` accessor yields a global variable, not a projection from `self`.
    // Diagnosing active applications is nonetheless a safe over-approximation.
    _modify { yield &global }
  }
}
@differentiable
func TF_1115_modifyNonSelfProjection(x: Float) -> Float {
  var result: Float = 0
  // Assignment below calls `Float.projection.modify`.
  result.projection = x
  return result
}

$ swift modify.swift
modify.swift:10:2: error: function is not differentiable
@differentiable
~^~~~~~~~~~~~~~
modify.swift:11:6: note: when differentiating this function definition
func TF_1115_modifyNonSelfProjection(x: Float) -> Float {
     ^
modify.swift:14:21: note: differentiation of coroutine calls is not yet supported
  result.projection = x
                    ^

[AutoDiff] Fix undiagnosed active `modify` accessor applications.

a782bffb

dan-zheng added tensorflow

dan-zheng requested a review from

marcrasi 5 years ago

dan-zheng requested a review from

rxwei 5 years ago

rxwei approved these changes on 2020-01-31

dan-zheng5 years ago

@swift-ci Please test tensorflow

dan-zheng5 years ago (edited 5 years ago)

The activity analysis changes cause new non-differentiability errors during TensorFlow module compilation:

/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/tensorflow-swift-apis/Sources/TensorFlow/Layers/Recurrent.swift:417:6: error: function is not differentiable
    @differentiable(wrt: (self, inputs))
    ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/tensorflow-swift-apis/Sources/TensorFlow/Layers/Recurrent.swift:418:17: note: when differentiating this function definition
    public func lastOutput(
                ^
/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/tensorflow-swift-apis/Sources/TensorFlow/Layers/Recurrent.swift:423:86: note: cannot differentiate through a non-differentiable result; do you want to use 'withoutDerivative(at:)'?
        return self(inputs, initialState: initialState)[withoutDerivative(at: inputs.count - 1)]
                                                                                     ^
/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/tensorflow-swift-apis/Sources/TensorFlow/Operators/Basic.swift:518:65: error: expression is not differentiable
        let batchIndices: Tensor<Index> = withoutDerivative(at: {
                                                                ^
/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/tensorflow-swift-apis/Sources/TensorFlow/Operators/Basic.swift:518:65: note: differentiated functions in '@inlinable' functions must be marked '@differentiable' or have a public '@derivative'; this is not possible with a closure, make a top-level function instead
        let batchIndices: Tensor<Index> = withoutDerivative(at: {
                                                                ^

Investigating now.

It would be nice if the cannot differentiate through a non-differentiable result error showed the name of the referenced Swift declaration (if one exists).

Edit: these errors were due to incorrect (unintentional) activity analysis changes. Fixed in 9779e05.

Revert incorrect activity analysis changes.

9779e052

dan-zheng5 years ago

@swift-ci Please clean test tensorflow

dan-zheng merged 3c40ac51 into tensorflow 5 years ago

dan-zheng deleted the TF-1115 branch 5 years ago

swift
[AutoDiff] Fix undiagnosed active `modify` accessor applications.
#29564

Merged

[AutoDiff] Fix undiagnosed active `modify` accessor applications. #29564

swift [AutoDiff] Fix undiagnosed active `modify` accessor applications. #29564 Merged

[AutoDiff] Fix undiagnosed active `modify` accessor applications. #29564

swift
[AutoDiff] Fix undiagnosed active `modify` accessor applications.
#29564

Merged