[Turbopack] reland refactor filesystem writes to an effect based system (#73294)
Reverting the revert https://github.com/vercel/next.js/pull/73287
see https://github.com/vercel/next.js/pull/72847
---
Quoting from the tokio::fs::File documentation:
> A file will not be closed immediately when it goes out of scope if there are any IO operations that have not yet completed. To ensure that a file is closed immediately when it is dropped, you should call flush before dropping it.
So that means we will return from write_to_disk before files are fully written. That was a bug before, but since with effects the write calls are executed much later. The manifest files are read directly after write_to_disk returns, so they are most likely to fail.
So adding `flush` to fix that.
---
Another bug with the effects implementation is that only the first `apply_effects` call waits for the effect execution to complete. If there are concurrent `apply_effects` on the same effects, the later one would return immediately.
Added some logic to handle that.